2023-text-tonality.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["# Задача **\"Классификация тональности текста\"**\n","\n","\n","---\n","\n","**Выполнил**: Егор Юмаев"],"metadata":{"id":"YId1qPMldmQ1"}},{"cell_type":"markdown","source":["## **1. Введение**\n","\n","---"],"metadata":{"id":"9bA1QY0ef9RT"}},{"cell_type":"markdown","source":["Неотъемлемой частью современного мира являются электронные площадки, сервисы, порталы, на которых пользователи могут оставлять свои отзывы. Тональность отзывов может варьироваться от доброжелательной до негативной.\n","\n","Ручной анализ комментариев пользователей может быть эффективен только при небольших количествах письменных откликов. При наличии большого потока сообщений необходимо внедрение автоматизированной системы определения тональности комментариев.\n","\n","Задача оценки тональности текста является задачей **классификации**, в которой положительному тексту может быть присвоено значение \"1\", негативному - \"0\". Признаки представлены словами из корпуса и их величинами TF-IDF для каждого конкретного текста сообщения."],"metadata":{"id":"T3_C8qq0d4fB"}},{"cell_type":"markdown","source":["## **2. Задание**\n","\n","\n","---"],"metadata":{"id":"okRmNxeCgHV5"}},{"cell_type":"markdown","source":["Требуется обучить логистическую регрессию для определения тональности текста.\n","\n","Исходные данные представлены двумя датасетами (обучающим и тестовым), включающими тексты комментариев пользователей в исходном и лемматизированном виде. Целевой бинарный признак 'positive' содержится в обучающем датасете.\n","\n","Предсказания необходимо сохранить в столбце positive и преобразовать в отдельный csv-файл, загружаемый в облачную систему со скрытыми правильными ответами 'positive' тестовой выборки.\n","\n","**Целевое ограничение**: значение метрики **accuracy** должно быть **> 0.62**."],"metadata":{"id":"mXpcC9uBgTRD"}},{"cell_type":"markdown","source":["## **3. Решение**\n","\n","---"],"metadata":{"id":"hm0IwNThhJqB"}},{"cell_type":"code","execution_count":1,"metadata":{"id":"ABpndDu_4m42","executionInfo":{"status":"ok","timestamp":1695444221956,"user_tz":-360,"elapsed":25214,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}},"colab":{"base_uri":"https://localhost:8080/"},"outputId":"2bec666c-f967-4758-f697-34009b902041"},"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/gdrive\n"]}],"source":["# загружаем необходимые библиотеки\n","import pandas as pd\n","import nltk\n","from nltk.corpus import stopwords as nltk_stopwords\n","from sklearn.feature_extraction.text import TfidfVectorizer\n","from sklearn.linear_model import LogisticRegression\n","from sklearn.metrics import accuracy_score\n","\n","# подключаем Google Drive для последующей загрузки на него\n","# файла с предсказаниями\n","from google.colab import drive\n","drive.mount('/content/gdrive')"]},{"cell_type":"code","source":["# загружаем zip-архив с обучающим и тестовым датасетами\n","! gdown 1XsMwM7t7FxGFN5-M74esbKOD9tF0vCk2\n","\n","# разархивируем загруженный архив\n","! unzip -o /content/tonality_of_the_text.zip\n","\n","# сохраняем датасеты в переменные\n","train = pd.read_csv('/content/tonality_of_the_text/tweets_lemm_train.csv')\n","test = pd.read_csv('/content/tonality_of_the_text/tweets_lemm_test.csv')\n","print('Datasets load from Google Drive storage')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"6zdZNJpvCLKB","executionInfo":{"status":"ok","timestamp":1695444223650,"user_tz":-360,"elapsed":1709,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}},"outputId":"93cdeafe-62b3-4362-8c8b-da29b977ae97"},"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["Downloading...\n","From: https://drive.google.com/uc?id=1XsMwM7t7FxGFN5-M74esbKOD9tF0vCk2\n","To: /content/tonality_of_the_text.zip\n","\r  0% 0.00/608k [00:00<?, ?B/s]\r100% 608k/608k [00:00<00:00, 89.6MB/s]\n","Archive:  /content/tonality_of_the_text.zip\n","  inflating: tonality_of_the_text/tweets_lemm_train.csv  \n","  inflating: tonality_of_the_text/tweets_lemm_test.csv  \n","Datasets load from Google Drive storage\n"]}]},{"cell_type":"markdown","source":["Проверим, загружены ли данные для анализа, для чего выведем первые пять строк датасетов train и test."],"metadata":{"id":"R4NStAALFfWs"}},{"cell_type":"code","source":["train.head()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"QtYAITsI4wVc","outputId":"99e3c783-bc15-43b5-eb37-aecccff8e953","executionInfo":{"status":"ok","timestamp":1695444223651,"user_tz":-360,"elapsed":31,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":3,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                                                text  positive  \\\n","0  @first_timee хоть я и школота, но поверь, у на...         1   \n","1  Да, все-таки он немного похож на него. Но мой ...         1   \n","2  RT @KatiaCheh: Ну ты идиотка) я испугалась за ...         1   \n","3  RT @digger2912: \"Кто то в углу сидит и погибае...         1   \n","4  @irina_dyshkant Вот что значит страшилка :D\\r\\...         1   \n","\n","                                           lemm_text  \n","0  хоть я и школотый но поверь у мы то же самый о...  \n","1  да весь таки он немного похожий на он но мой м...  \n","2                  ну ты идиотка я испугаться за ты   \n","3  кто то в угол сидеть и погибать от голод а мы ...  \n","4  вот что значит страшилка но блин посмотреть ве...  "],"text/html":["\n","  <div id=\"df-c6454093-5ded-4fba-9d81-5559ac41b7df\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>positive</th>\n","      <th>lemm_text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>@first_timee хоть я и школота, но поверь, у на...</td>\n","      <td>1</td>\n","      <td>хоть я и школотый но поверь у мы то же самый о...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Да, все-таки он немного похож на него. Но мой ...</td>\n","      <td>1</td>\n","      <td>да весь таки он немного похожий на он но мой м...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>RT @KatiaCheh: Ну ты идиотка) я испугалась за ...</td>\n","      <td>1</td>\n","      <td>ну ты идиотка я испугаться за ты</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>RT @digger2912: \"Кто то в углу сидит и погибае...</td>\n","      <td>1</td>\n","      <td>кто то в угол сидеть и погибать от голод а мы ...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>@irina_dyshkant Вот что значит страшилка :D\\r\\...</td>\n","      <td>1</td>\n","      <td>вот что значит страшилка но блин посмотреть ве...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c6454093-5ded-4fba-9d81-5559ac41b7df')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-c6454093-5ded-4fba-9d81-5559ac41b7df button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-c6454093-5ded-4fba-9d81-5559ac41b7df');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-4c7491e6-fd3c-404c-a600-7eb54a46e65a\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4c7491e6-fd3c-404c-a600-7eb54a46e65a')\"\n","            title=\"Suggest charts.\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-4c7491e6-fd3c-404c-a600-7eb54a46e65a button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":3}]},{"cell_type":"code","source":["test.head()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"g4TLtHVs4wZA","outputId":"65eb9dab-08a3-421a-e680-e362c5d32d13","executionInfo":{"status":"ok","timestamp":1695444223651,"user_tz":-360,"elapsed":26,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":4,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                                                text  \\\n","0  RT @tiredfennel: если криса так интересуют дет...   \n","1  @xsealord по 200 руб. в месяц можно разместить...   \n","2  @haosANDlaw @Etishkindyx учитывая, что сейчас ...   \n","3  Товарищ    :) Но я никак не могу отдельно не о...   \n","4  RT @BodyaNick: Квн был отличный !) Оооочень по...   \n","\n","                                           lemm_text  \n","0  если крис так интересовать ребёнок то либо они...  \n","1  по рубль в месяц можно разместить ссылка на те...  \n","2  учитывать что сейчас преобладать один половина...  \n","3  товарищ но я никак не мочь отдельно не отметит...  \n","4  квн быть отличный оооочень понравиться женский...  "],"text/html":["\n","  <div id=\"df-9e8875bb-b60b-4a14-8e62-de05de903b5b\" class=\"colab-df-container\">\n","    <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>text</th>\n","      <th>lemm_text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>RT @tiredfennel: если криса так интересуют дет...</td>\n","      <td>если крис так интересовать ребёнок то либо они...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>@xsealord по 200 руб. в месяц можно разместить...</td>\n","      <td>по рубль в месяц можно разместить ссылка на те...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>@haosANDlaw @Etishkindyx учитывая, что сейчас ...</td>\n","      <td>учитывать что сейчас преобладать один половина...</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Товарищ    :) Но я никак не могу отдельно не о...</td>\n","      <td>товарищ но я никак не мочь отдельно не отметит...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>RT @BodyaNick: Квн был отличный !) Оооочень по...</td>\n","      <td>квн быть отличный оооочень понравиться женский...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","    <div class=\"colab-df-buttons\">\n","\n","  <div class=\"colab-df-container\">\n","    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9e8875bb-b60b-4a14-8e62-de05de903b5b')\"\n","            title=\"Convert this dataframe to an interactive table.\"\n","            style=\"display:none;\">\n","\n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n","    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n","  </svg>\n","    </button>\n","\n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    .colab-df-buttons div {\n","      margin-bottom: 4px;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","    <script>\n","      const buttonEl =\n","        document.querySelector('#df-9e8875bb-b60b-4a14-8e62-de05de903b5b button.colab-df-convert');\n","      buttonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","      async function convertToInteractive(key) {\n","        const element = document.querySelector('#df-9e8875bb-b60b-4a14-8e62-de05de903b5b');\n","        const dataTable =\n","          await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                    [key], {});\n","        if (!dataTable) return;\n","\n","        const docLinkHtml = 'Like what you see? Visit the ' +\n","          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","          + ' to learn more about interactive tables.';\n","        element.innerHTML = '';\n","        dataTable['output_type'] = 'display_data';\n","        await google.colab.output.renderOutput(dataTable, element);\n","        const docLink = document.createElement('div');\n","        docLink.innerHTML = docLinkHtml;\n","        element.appendChild(docLink);\n","      }\n","    </script>\n","  </div>\n","\n","\n","<div id=\"df-3e6e0cd5-9ebb-4d84-9375-fd9a84209594\">\n","  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3e6e0cd5-9ebb-4d84-9375-fd9a84209594')\"\n","            title=\"Suggest charts.\"\n","            style=\"display:none;\">\n","\n","<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","     width=\"24px\">\n","    <g>\n","        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n","    </g>\n","</svg>\n","  </button>\n","\n","<style>\n","  .colab-df-quickchart {\n","      --bg-color: #E8F0FE;\n","      --fill-color: #1967D2;\n","      --hover-bg-color: #E2EBFA;\n","      --hover-fill-color: #174EA6;\n","      --disabled-fill-color: #AAA;\n","      --disabled-bg-color: #DDD;\n","  }\n","\n","  [theme=dark] .colab-df-quickchart {\n","      --bg-color: #3B4455;\n","      --fill-color: #D2E3FC;\n","      --hover-bg-color: #434B5C;\n","      --hover-fill-color: #FFFFFF;\n","      --disabled-bg-color: #3B4455;\n","      --disabled-fill-color: #666;\n","  }\n","\n","  .colab-df-quickchart {\n","    background-color: var(--bg-color);\n","    border: none;\n","    border-radius: 50%;\n","    cursor: pointer;\n","    display: none;\n","    fill: var(--fill-color);\n","    height: 32px;\n","    padding: 0;\n","    width: 32px;\n","  }\n","\n","  .colab-df-quickchart:hover {\n","    background-color: var(--hover-bg-color);\n","    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n","    fill: var(--button-hover-fill-color);\n","  }\n","\n","  .colab-df-quickchart-complete:disabled,\n","  .colab-df-quickchart-complete:disabled:hover {\n","    background-color: var(--disabled-bg-color);\n","    fill: var(--disabled-fill-color);\n","    box-shadow: none;\n","  }\n","\n","  .colab-df-spinner {\n","    border: 2px solid var(--fill-color);\n","    border-color: transparent;\n","    border-bottom-color: var(--fill-color);\n","    animation:\n","      spin 1s steps(1) infinite;\n","  }\n","\n","  @keyframes spin {\n","    0% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","      border-left-color: var(--fill-color);\n","    }\n","    20% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    30% {\n","      border-color: transparent;\n","      border-left-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","      border-right-color: var(--fill-color);\n","    }\n","    40% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-top-color: var(--fill-color);\n","    }\n","    60% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","    }\n","    80% {\n","      border-color: transparent;\n","      border-right-color: var(--fill-color);\n","      border-bottom-color: var(--fill-color);\n","    }\n","    90% {\n","      border-color: transparent;\n","      border-bottom-color: var(--fill-color);\n","    }\n","  }\n","</style>\n","\n","  <script>\n","    async function quickchart(key) {\n","      const quickchartButtonEl =\n","        document.querySelector('#' + key + ' button');\n","      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n","      quickchartButtonEl.classList.add('colab-df-spinner');\n","      try {\n","        const charts = await google.colab.kernel.invokeFunction(\n","            'suggestCharts', [key], {});\n","      } catch (error) {\n","        console.error('Error during call to suggestCharts:', error);\n","      }\n","      quickchartButtonEl.classList.remove('colab-df-spinner');\n","      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n","    }\n","    (() => {\n","      let quickchartButtonEl =\n","        document.querySelector('#df-3e6e0cd5-9ebb-4d84-9375-fd9a84209594 button');\n","      quickchartButtonEl.style.display =\n","        google.colab.kernel.accessAllowed ? 'block' : 'none';\n","    })();\n","  </script>\n","</div>\n","    </div>\n","  </div>\n"]},"metadata":{},"execution_count":4}]},{"cell_type":"code","source":["# выведем размеры загруженных датасетов\n","train.shape, test.shape"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2V8XBm6h4wcj","outputId":"b00fcb8b-2146-401d-f6da-a261de039ada","executionInfo":{"status":"ok","timestamp":1695444223652,"user_tz":-360,"elapsed":24,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":5,"outputs":[{"output_type":"execute_result","data":{"text/plain":["((5000, 3), (3000, 2))"]},"metadata":{},"execution_count":5}]},{"cell_type":"markdown","source":["Данные успешно загружены."],"metadata":{"id":"sj-5MxXiF0XP"}},{"cell_type":"markdown","source":["Чтобы алгоритм смог определить тональность текста, его необходимо обучить на корпусе - наборе текстов, в котором эмоции и ключевые слова уже размечены. Для этого необходимо перевести тексты в стандартный для Python формат - кодировку Unicode."],"metadata":{"id":"FiWyCSxQGTro"}},{"cell_type":"code","source":["# создаем корпус слов из лемматизированных сообщений\n","corpus_train = train['lemm_text'].values.astype('U')"],"metadata":{"id":"5hHbNeN-BBgy","executionInfo":{"status":"ok","timestamp":1695444223653,"user_tz":-360,"elapsed":18,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":6,"outputs":[]},{"cell_type":"code","source":["# создаем корпус слов из лемматизированных сообщений\n","corpus_test = test['lemm_text'].values.astype('U')"],"metadata":{"id":"kOxoHBMfBp4V","executionInfo":{"status":"ok","timestamp":1695444223654,"user_tz":-360,"elapsed":18,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":["Тексты необходимо очистить от стоп-слов - слов, не несущих смысловой нагрузки. Для этого необходим пакет \"stopwords\" с аргументом \"russian\" (стоп-словами, характерными для русского языка)."],"metadata":{"id":"pa6Wo7doHmuy"}},{"cell_type":"code","source":["# загружаем стоп-слова\n","nltk.download('stopwords')\n","stopwords = list(set(nltk_stopwords.words('russian')))"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"iyfnw8K64wjI","outputId":"0965c3f2-3c92-4ede-d8df-bb2805cfa02d","executionInfo":{"status":"ok","timestamp":1695444224109,"user_tz":-360,"elapsed":473,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":8,"outputs":[{"output_type":"stream","name":"stderr","text":["[nltk_data] Downloading package stopwords to /root/nltk_data...\n","[nltk_data]   Unzipping corpora/stopwords.zip.\n"]}]},{"cell_type":"markdown","source":["Оценка важности слова определяется величиной TF-IDF:\n","\n"," * TF отвечает за количество упоминаний слова в отдельном тексте\n","\n"," * IDF отражает частоту употребления слова во всем корпусе\n","\n","Большая величина TF-IDF гооврит об уникальности слова в тексте по отношению к корпусу. Чем чаще оно встречается в конкретном корпусе и реже в остальных, тем выше значение TF-IDF."],"metadata":{"id":"AXWzo9WtIk0w"}},{"cell_type":"code","source":["# объявим счетчик величин TF-IDF\n","count_tf_idf = TfidfVectorizer(stop_words = stopwords)"],"metadata":{"id":"Y3sJsRu74wme","executionInfo":{"status":"ok","timestamp":1695444224109,"user_tz":-360,"elapsed":12,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":9,"outputs":[]},{"cell_type":"code","source":["# выведем размер корпусов до применения счетчика\n","corpus_train.shape, corpus_test.shape"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"YMae2FtZCZEQ","outputId":"e0387207-3b3b-4bd2-b959-addb615560e3","executionInfo":{"status":"ok","timestamp":1695444224110,"user_tz":-360,"elapsed":12,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["((5000,), (3000,))"]},"metadata":{},"execution_count":10}]},{"cell_type":"code","source":["# применим счетчик к лемматизированным текстам обучающей выборки\n","corpus_train = count_tf_idf.fit_transform(corpus_train)"],"metadata":{"id":"CrcEIcD9CZHS","executionInfo":{"status":"ok","timestamp":1695444224111,"user_tz":-360,"elapsed":11,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":11,"outputs":[]},{"cell_type":"code","source":["# применим счетчик к лемматизированным текстам тестовой выборки\n","corpus_test = count_tf_idf.transform(corpus_test)"],"metadata":{"id":"A6B-Dhe0Dxhl","executionInfo":{"status":"ok","timestamp":1695444224112,"user_tz":-360,"elapsed":12,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":12,"outputs":[]},{"cell_type":"code","source":["# выведем размер корпусов текстов после применения счетчика\n","corpus_train.shape, corpus_test.shape"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"IkcZGLjKCZKM","outputId":"694c7998-bcb6-4a27-a208-d28b1d081b77","executionInfo":{"status":"ok","timestamp":1695444224112,"user_tz":-360,"elapsed":12,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":13,"outputs":[{"output_type":"execute_result","data":{"text/plain":["((5000, 9737), (3000, 9737))"]},"metadata":{},"execution_count":13}]},{"cell_type":"code","source":["# объявляем модель\n","model = LogisticRegression(random_state = 1)"],"metadata":{"id":"9nPHT_JE78eg","executionInfo":{"status":"ok","timestamp":1695444224112,"user_tz":-360,"elapsed":9,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":14,"outputs":[]},{"cell_type":"code","source":["# обучаем модель\n","model.fit(corpus_train, train['positive'])"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":74},"id":"LMkgbiD8Cxcn","outputId":"213b0d0d-92a9-4098-a461-e2d666e121d1","executionInfo":{"status":"ok","timestamp":1695444224508,"user_tz":-360,"elapsed":405,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":15,"outputs":[{"output_type":"execute_result","data":{"text/plain":["LogisticRegression(random_state=1)"],"text/html":["<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(random_state=1)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(random_state=1)</pre></div></div></div></div></div>"]},"metadata":{},"execution_count":15}]},{"cell_type":"code","source":["# получаем предсказания модели на корпусе текстов обучающей выборки\n","predict_train = model.predict(corpus_train)"],"metadata":{"id":"fL13lezCCxfa","executionInfo":{"status":"ok","timestamp":1695444224509,"user_tz":-360,"elapsed":19,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":16,"outputs":[]},{"cell_type":"code","source":["# вычислим точность предсказаний на обучающей выборке\n","accuracy_train = accuracy_score(train['positive'], predict_train)\n","print(f'accuracy на обучающей выборке:\\n {accuracy_train}')"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"k8seECVTCZNS","outputId":"695ce067-f5fd-4dca-a60e-787131f49409","executionInfo":{"status":"ok","timestamp":1695444224510,"user_tz":-360,"elapsed":19,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":17,"outputs":[{"output_type":"stream","name":"stdout","text":["accuracy на обучающей выборке:\n"," 0.8984\n"]}]},{"cell_type":"code","source":["# получим предсказания модели на корпусе текстов тестовой выборки\n","predictions = pd.Series(model.predict(corpus_test), name = 'positive')"],"metadata":{"id":"_cBMrBpOOdcw","executionInfo":{"status":"ok","timestamp":1695444224510,"user_tz":-360,"elapsed":15,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":18,"outputs":[]},{"cell_type":"code","source":["# выведем первые 10 строк полученных предсказаний\n","predictions.head()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xVdvhByPAFwY","outputId":"30bb10b3-00a1-482e-f8f0-8f84d3d2979c","executionInfo":{"status":"ok","timestamp":1695444224511,"user_tz":-360,"elapsed":15,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0    1\n","1    0\n","2    0\n","3    0\n","4    1\n","Name: positive, dtype: int64"]},"metadata":{},"execution_count":19}]},{"cell_type":"code","source":["# загружаем файл с предсказаниями на Google Drive\n","predictions.to_csv('/content/gdrive/My Drive/predictions.csv', index = False)"],"metadata":{"id":"VhOtNHOk7vwf","executionInfo":{"status":"ok","timestamp":1695444224511,"user_tz":-360,"elapsed":10,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":20,"outputs":[]},{"cell_type":"code","source":[],"metadata":{"id":"--FiDbmS6pWl","executionInfo":{"status":"ok","timestamp":1695444224512,"user_tz":-360,"elapsed":9,"user":{"displayName":"Egor Yumaev","userId":"00994742406983626184"}}},"execution_count":20,"outputs":[]}]}