s12_02_web_scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "857001a9",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "# IT Academy - Data Science Itinerary\n",
    "\n",
    "\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69a5582",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "In this project we are going to use 3 of the most used libraries to do web scraping: **Beautifulsoup**, **Selenium** and **Scrapy**.\n",
    "\n",
    "$~$\n",
    "\n",
    "*This project is divided into three parts, in the first two we are going to use beatifulsoup and selenium to extract information from the Madrid Stock Exchange website. The intention is to test each of these libraries for the same task and see how efficient they are.*\n",
    "\n",
    "$~$\n",
    "\n",
    "*In the third part we are going to use the scrapy library to obtain 190 articles from the website of the newspaper El Pais (English version).*\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad6e86cc",
   "metadata": {},
   "source": [
    "## S12 T12: Web Scraping\n",
    "___\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "343f71ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing libraries\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "from selenium.common.exceptions import TimeoutException\n",
    "from time import sleep\n",
    "\n",
    "\n",
    "import scrapy\n",
    "from scrapy.crawler import CrawlerProcess\n",
    "import os\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7557656a",
   "metadata": {},
   "source": [
    "$~$\n",
    "___\n",
    "####  Exercise 1\n",
    "\n",
    "$~$\n",
    "\n",
    "Scrape a page on the [Madrid Stock Exchange](https://www.bolsamadrid.es) using BeautifulSoup and Selenium.\n",
    "\n",
    "$~$\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a530b0c4",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "In this first part of the project we are going to scrap the website of the Madrid Stock Exchange. Specifically we are going to extract the information of the [prices of the session](https://www.bolsamadrid.es/ing/aspx/Mercados/Precios.aspx?indice=ESI100000000) of the day. We want to store this information in a Pandas dataframe. For do it we are going to use two libraries: First using the **Beautifulsoup** library, then  using the **Selemiun library**.\n",
    "\n",
    "$~$\n",
    "\n",
    "In the following image we will see the **keys** we have used to carry out the exercise, on the one hand, the **table id** and on the other the **\"X_Path\"** of the table. From this information we have been able to obtain all the information from the table\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c339bdf",
   "metadata": {},
   "source": [
    "+ note here the **Id** and **X_path** we are going to use to scrap the table\n",
    "<img src=\"image.png\" alt=\"alt_text\" align=\"center\" width=\"650\" height=\"320\" />\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b98a74be",
   "metadata": {},
   "source": [
    "[](./image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "24f74a7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#define the url to scrap\n",
    "url = \"https://www.bolsamadrid.es/ing/aspx/Mercados/Precios.aspx?indice=ESI100000000\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1feccffc",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "+ Scraping using BeatifulSoup:\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "327e5c6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Last</th>\n",
       "      <th>% Dif.</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Turnover (€ Thousands)</th>\n",
       "      <th>Date</th>\n",
       "      <th>Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ACCIONA</td>\n",
       "      <td>189.2000</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>190.0000</td>\n",
       "      <td>187.4000</td>\n",
       "      <td>20,137</td>\n",
       "      <td>3,808.69</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ACERINOX</td>\n",
       "      <td>11.6800</td>\n",
       "      <td>-0.34</td>\n",
       "      <td>11.7600</td>\n",
       "      <td>11.6050</td>\n",
       "      <td>823,017</td>\n",
       "      <td>9,619.55</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ACS</td>\n",
       "      <td>26.7900</td>\n",
       "      <td>-0.22</td>\n",
       "      <td>26.9100</td>\n",
       "      <td>26.6600</td>\n",
       "      <td>144,000</td>\n",
       "      <td>3,862.44</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AENA</td>\n",
       "      <td>141.6000</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>142.1000</td>\n",
       "      <td>141.3500</td>\n",
       "      <td>20,318</td>\n",
       "      <td>2,879.65</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALMIRALL</td>\n",
       "      <td>10.3900</td>\n",
       "      <td>0.39</td>\n",
       "      <td>10.8200</td>\n",
       "      <td>10.3600</td>\n",
       "      <td>403,502</td>\n",
       "      <td>4,256.16</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:21:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>AMADEUS</td>\n",
       "      <td>57.2000</td>\n",
       "      <td>-0.97</td>\n",
       "      <td>57.6400</td>\n",
       "      <td>56.7800</td>\n",
       "      <td>84,981</td>\n",
       "      <td>4,863.99</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ARCELORMIT.</td>\n",
       "      <td>30.6950</td>\n",
       "      <td>-0.34</td>\n",
       "      <td>30.8150</td>\n",
       "      <td>30.4500</td>\n",
       "      <td>79,819</td>\n",
       "      <td>2,443.79</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>B.SANTANDER</td>\n",
       "      <td>3.0150</td>\n",
       "      <td>-0.76</td>\n",
       "      <td>3.0420</td>\n",
       "      <td>3.0090</td>\n",
       "      <td>5,023,780</td>\n",
       "      <td>15,185.24</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BA.SABADELL</td>\n",
       "      <td>0.8510</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.8560</td>\n",
       "      <td>0.8452</td>\n",
       "      <td>7,015,644</td>\n",
       "      <td>5,970.51</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BANKINTER</td>\n",
       "      <td>5.9320</td>\n",
       "      <td>-0.07</td>\n",
       "      <td>5.9640</td>\n",
       "      <td>5.9100</td>\n",
       "      <td>319,594</td>\n",
       "      <td>1,899.05</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>BBVA</td>\n",
       "      <td>5.0700</td>\n",
       "      <td>-0.67</td>\n",
       "      <td>5.1070</td>\n",
       "      <td>5.0630</td>\n",
       "      <td>1,377,096</td>\n",
       "      <td>7,003.15</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>CAIXABANK</td>\n",
       "      <td>3.4720</td>\n",
       "      <td>0.26</td>\n",
       "      <td>3.4930</td>\n",
       "      <td>3.4350</td>\n",
       "      <td>5,227,626</td>\n",
       "      <td>18,176.22</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>CELLNEX</td>\n",
       "      <td>41.6500</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>41.7900</td>\n",
       "      <td>41.2000</td>\n",
       "      <td>103,162</td>\n",
       "      <td>4,279.20</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>CIE AUTOMOT.</td>\n",
       "      <td>24.6400</td>\n",
       "      <td>0.90</td>\n",
       "      <td>24.7600</td>\n",
       "      <td>24.2800</td>\n",
       "      <td>63,828</td>\n",
       "      <td>1,569.42</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ENAGAS</td>\n",
       "      <td>22.0200</td>\n",
       "      <td>1.47</td>\n",
       "      <td>22.0600</td>\n",
       "      <td>21.6400</td>\n",
       "      <td>197,521</td>\n",
       "      <td>4,328.12</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ENDESA</td>\n",
       "      <td>20.8000</td>\n",
       "      <td>0.63</td>\n",
       "      <td>20.8400</td>\n",
       "      <td>20.5300</td>\n",
       "      <td>187,176</td>\n",
       "      <td>3,879.64</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>FERROVIAL</td>\n",
       "      <td>25.0900</td>\n",
       "      <td>0.97</td>\n",
       "      <td>25.2400</td>\n",
       "      <td>24.8500</td>\n",
       "      <td>227,616</td>\n",
       "      <td>5,716.42</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>FLUIDRA</td>\n",
       "      <td>24.8800</td>\n",
       "      <td>-0.88</td>\n",
       "      <td>25.2200</td>\n",
       "      <td>24.7600</td>\n",
       "      <td>110,688</td>\n",
       "      <td>2,759.17</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>GRIFOLS CL.A</td>\n",
       "      <td>18.9250</td>\n",
       "      <td>-0.97</td>\n",
       "      <td>19.1000</td>\n",
       "      <td>18.8350</td>\n",
       "      <td>134,784</td>\n",
       "      <td>2,556.24</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>IAG</td>\n",
       "      <td>1.4540</td>\n",
       "      <td>-1.46</td>\n",
       "      <td>1.4830</td>\n",
       "      <td>1.4500</td>\n",
       "      <td>8,114,181</td>\n",
       "      <td>11,878.24</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>IBERDROLA</td>\n",
       "      <td>10.8600</td>\n",
       "      <td>-0.37</td>\n",
       "      <td>10.9600</td>\n",
       "      <td>10.8150</td>\n",
       "      <td>4,950,247</td>\n",
       "      <td>53,706.68</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>INDITEX</td>\n",
       "      <td>22.0000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>22.0500</td>\n",
       "      <td>21.8000</td>\n",
       "      <td>1,013,878</td>\n",
       "      <td>22,248.44</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>INDRA A</td>\n",
       "      <td>10.6600</td>\n",
       "      <td>-0.56</td>\n",
       "      <td>10.7100</td>\n",
       "      <td>10.6000</td>\n",
       "      <td>132,462</td>\n",
       "      <td>1,412.26</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:21:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>INM.COLONIAL</td>\n",
       "      <td>7.5850</td>\n",
       "      <td>1.47</td>\n",
       "      <td>7.6200</td>\n",
       "      <td>7.4350</td>\n",
       "      <td>208,701</td>\n",
       "      <td>1,575.78</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>MAPFRE</td>\n",
       "      <td>1.7560</td>\n",
       "      <td>0.23</td>\n",
       "      <td>1.7720</td>\n",
       "      <td>1.7530</td>\n",
       "      <td>2,436,838</td>\n",
       "      <td>4,305.47</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:21:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>MELIA HOTELS</td>\n",
       "      <td>7.6200</td>\n",
       "      <td>1.40</td>\n",
       "      <td>7.6700</td>\n",
       "      <td>7.5300</td>\n",
       "      <td>237,370</td>\n",
       "      <td>1,806.83</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>MERLIN</td>\n",
       "      <td>10.5400</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10.5900</td>\n",
       "      <td>10.5200</td>\n",
       "      <td>186,757</td>\n",
       "      <td>1,972.88</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>NATURGY</td>\n",
       "      <td>28.4600</td>\n",
       "      <td>0.42</td>\n",
       "      <td>28.5600</td>\n",
       "      <td>28.0900</td>\n",
       "      <td>47,586</td>\n",
       "      <td>1,352.59</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>PHARMA MAR</td>\n",
       "      <td>69.2800</td>\n",
       "      <td>-1.84</td>\n",
       "      <td>70.4000</td>\n",
       "      <td>69.1600</td>\n",
       "      <td>25,092</td>\n",
       "      <td>1,743.68</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>R.E.C.</td>\n",
       "      <td>19.1100</td>\n",
       "      <td>0.45</td>\n",
       "      <td>19.1600</td>\n",
       "      <td>18.9550</td>\n",
       "      <td>133,476</td>\n",
       "      <td>2,544.16</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>REPSOL</td>\n",
       "      <td>15.8800</td>\n",
       "      <td>3.02</td>\n",
       "      <td>16.0450</td>\n",
       "      <td>15.3900</td>\n",
       "      <td>2,760,539</td>\n",
       "      <td>43,345.48</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>ROVI</td>\n",
       "      <td>60.0000</td>\n",
       "      <td>-0.99</td>\n",
       "      <td>60.5000</td>\n",
       "      <td>59.8000</td>\n",
       "      <td>27,131</td>\n",
       "      <td>1,631.34</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:18:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>SIEMENS GAME</td>\n",
       "      <td>18.0250</td>\n",
       "      <td>-0.06</td>\n",
       "      <td>18.0500</td>\n",
       "      <td>18.0050</td>\n",
       "      <td>1,549,486</td>\n",
       "      <td>27,936.40</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>SOLARIA</td>\n",
       "      <td>22.7700</td>\n",
       "      <td>2.71</td>\n",
       "      <td>22.8200</td>\n",
       "      <td>22.0900</td>\n",
       "      <td>182,454</td>\n",
       "      <td>4,110.71</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>TELEFONICA</td>\n",
       "      <td>4.6380</td>\n",
       "      <td>-0.94</td>\n",
       "      <td>4.6750</td>\n",
       "      <td>4.6250</td>\n",
       "      <td>2,346,281</td>\n",
       "      <td>10,892.69</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Name      Last % Dif.      High       Low     Volume  \\\n",
       "0        ACCIONA  189.2000  -0.16  190.0000  187.4000     20,137   \n",
       "1       ACERINOX   11.6800  -0.34   11.7600   11.6050    823,017   \n",
       "2            ACS   26.7900  -0.22   26.9100   26.6600    144,000   \n",
       "3           AENA  141.6000  -0.32  142.1000  141.3500     20,318   \n",
       "4       ALMIRALL   10.3900   0.39   10.8200   10.3600    403,502   \n",
       "5        AMADEUS   57.2000  -0.97   57.6400   56.7800     84,981   \n",
       "6    ARCELORMIT.   30.6950  -0.34   30.8150   30.4500     79,819   \n",
       "7    B.SANTANDER    3.0150  -0.76    3.0420    3.0090  5,023,780   \n",
       "8    BA.SABADELL    0.8510   0.07    0.8560    0.8452  7,015,644   \n",
       "9      BANKINTER    5.9320  -0.07    5.9640    5.9100    319,594   \n",
       "10          BBVA    5.0700  -0.67    5.1070    5.0630  1,377,096   \n",
       "11     CAIXABANK    3.4720   0.26    3.4930    3.4350  5,227,626   \n",
       "12       CELLNEX   41.6500  -0.10   41.7900   41.2000    103,162   \n",
       "13  CIE AUTOMOT.   24.6400   0.90   24.7600   24.2800     63,828   \n",
       "14        ENAGAS   22.0200   1.47   22.0600   21.6400    197,521   \n",
       "15        ENDESA   20.8000   0.63   20.8400   20.5300    187,176   \n",
       "16     FERROVIAL   25.0900   0.97   25.2400   24.8500    227,616   \n",
       "17       FLUIDRA   24.8800  -0.88   25.2200   24.7600    110,688   \n",
       "18  GRIFOLS CL.A   18.9250  -0.97   19.1000   18.8350    134,784   \n",
       "19           IAG    1.4540  -1.46    1.4830    1.4500  8,114,181   \n",
       "20     IBERDROLA   10.8600  -0.37   10.9600   10.8150  4,950,247   \n",
       "21       INDITEX   22.0000   0.00   22.0500   21.8000  1,013,878   \n",
       "22       INDRA A   10.6600  -0.56   10.7100   10.6000    132,462   \n",
       "23  INM.COLONIAL    7.5850   1.47    7.6200    7.4350    208,701   \n",
       "24        MAPFRE    1.7560   0.23    1.7720    1.7530  2,436,838   \n",
       "25  MELIA HOTELS    7.6200   1.40    7.6700    7.5300    237,370   \n",
       "26        MERLIN   10.5400   0.00   10.5900   10.5200    186,757   \n",
       "27       NATURGY   28.4600   0.42   28.5600   28.0900     47,586   \n",
       "28    PHARMA MAR   69.2800  -1.84   70.4000   69.1600     25,092   \n",
       "29        R.E.C.   19.1100   0.45   19.1600   18.9550    133,476   \n",
       "30        REPSOL   15.8800   3.02   16.0450   15.3900  2,760,539   \n",
       "31          ROVI   60.0000  -0.99   60.5000   59.8000     27,131   \n",
       "32  SIEMENS GAME   18.0250  -0.06   18.0500   18.0050  1,549,486   \n",
       "33       SOLARIA   22.7700   2.71   22.8200   22.0900    182,454   \n",
       "34    TELEFONICA    4.6380  -0.94    4.6750    4.6250  2,346,281   \n",
       "\n",
       "   Turnover (€ Thousands)        Date      Time  \n",
       "0                3,808.69  07/06/2022  12:25:10  \n",
       "1                9,619.55  07/06/2022  12:25:15  \n",
       "2                3,862.44  07/06/2022  12:24:38  \n",
       "3                2,879.65  07/06/2022  12:24:57  \n",
       "4                4,256.16  07/06/2022  12:21:42  \n",
       "5                4,863.99  07/06/2022  12:24:53  \n",
       "6                2,443.79  07/06/2022  12:25:05  \n",
       "7               15,185.24  07/06/2022  12:25:17  \n",
       "8                5,970.51  07/06/2022  12:25:06  \n",
       "9                1,899.05  07/06/2022  12:25:06  \n",
       "10               7,003.15  07/06/2022  12:25:06  \n",
       "11              18,176.22  07/06/2022  12:25:03  \n",
       "12               4,279.20  07/06/2022  12:25:01  \n",
       "13               1,569.42  07/06/2022  12:23:14  \n",
       "14               4,328.12  07/06/2022  12:24:45  \n",
       "15               3,879.64  07/06/2022  12:25:06  \n",
       "16               5,716.42  07/06/2022  12:25:07  \n",
       "17               2,759.17  07/06/2022  12:25:16  \n",
       "18               2,556.24  07/06/2022  12:25:11  \n",
       "19              11,878.24  07/06/2022  12:25:01  \n",
       "20              53,706.68  07/06/2022  12:25:15  \n",
       "21              22,248.44  07/06/2022  12:25:09  \n",
       "22               1,412.26  07/06/2022  12:21:07  \n",
       "23               1,575.78  07/06/2022  12:24:15  \n",
       "24               4,305.47  07/06/2022  12:21:35  \n",
       "25               1,806.83  07/06/2022  12:23:32  \n",
       "26               1,972.88  07/06/2022  12:23:52  \n",
       "27               1,352.59  07/06/2022  12:23:49  \n",
       "28               1,743.68  07/06/2022  12:23:58  \n",
       "29               2,544.16  07/06/2022  12:24:55  \n",
       "30              43,345.48  07/06/2022  12:24:57  \n",
       "31               1,631.34  07/06/2022  12:18:50  \n",
       "32              27,936.40  07/06/2022  12:25:02  \n",
       "33               4,110.71  07/06/2022  12:24:25  \n",
       "34              10,892.69  07/06/2022  12:25:18  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#request the url \n",
    "web = requests.get(url)\n",
    "#init soup \n",
    "soup = BeautifulSoup(web.content,\"html.parser\")\n",
    "#find the table we want to scrap\n",
    "tables = soup.findChildren(\"table\", { \"id\" :\"ctl00_Contenido_tblAcciones\" })\n",
    "#limit table\n",
    "my_table = tables[0]\n",
    "#find rows in the table\n",
    "rows = my_table.findChildren(['tbody', 'tr'])\n",
    "#find names of the columns:\n",
    "names = my_table.findChildren(['tbody', 'th'])\n",
    "#keep columns names\n",
    "columns_name = []\n",
    "\n",
    "for name in names:\n",
    "    columns_name.append(name.text)\n",
    "\n",
    "#define empy dataframe with columns_names\n",
    "df1 = pd.DataFrame(columns=columns_name)\n",
    "\n",
    "#add values to our dataset\n",
    "for row in rows:\n",
    "    cols = []\n",
    "    for x in row.findAll(\"td\"):  \n",
    "      cols.append(x.text)\n",
    "    if len(cols)!=0:\n",
    "      df1.loc[len(df1.index)] = cols\n",
    "#show the table\n",
    "display(df1)\n",
    "\n",
    "#save to cvs\n",
    "df1.to_csv('data1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f24e895",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "+ Scraping using Selenium:\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a767ef84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Last</th>\n",
       "      <th>% Dif.</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Turnover (€ Thousands)</th>\n",
       "      <th>Date</th>\n",
       "      <th>Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ACCIONA</td>\n",
       "      <td>189.2000</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>190.0000</td>\n",
       "      <td>187.4000</td>\n",
       "      <td>20,137</td>\n",
       "      <td>3,808.69</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ACERINOX</td>\n",
       "      <td>11.6800</td>\n",
       "      <td>-0.34</td>\n",
       "      <td>11.7600</td>\n",
       "      <td>11.6050</td>\n",
       "      <td>823,017</td>\n",
       "      <td>9,619.55</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ACS</td>\n",
       "      <td>26.7900</td>\n",
       "      <td>-0.22</td>\n",
       "      <td>26.9100</td>\n",
       "      <td>26.6600</td>\n",
       "      <td>144,000</td>\n",
       "      <td>3,862.44</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AENA</td>\n",
       "      <td>141.6000</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>142.1000</td>\n",
       "      <td>141.3500</td>\n",
       "      <td>20,318</td>\n",
       "      <td>2,879.65</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALMIRALL</td>\n",
       "      <td>10.3900</td>\n",
       "      <td>0.39</td>\n",
       "      <td>10.8200</td>\n",
       "      <td>10.3600</td>\n",
       "      <td>403,502</td>\n",
       "      <td>4,256.16</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:21:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>AMADEUS</td>\n",
       "      <td>57.2000</td>\n",
       "      <td>-0.97</td>\n",
       "      <td>57.6400</td>\n",
       "      <td>56.7800</td>\n",
       "      <td>84,981</td>\n",
       "      <td>4,863.99</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ARCELORMIT.</td>\n",
       "      <td>30.6950</td>\n",
       "      <td>-0.34</td>\n",
       "      <td>30.8150</td>\n",
       "      <td>30.4500</td>\n",
       "      <td>79,819</td>\n",
       "      <td>2,443.79</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>B.SANTANDER</td>\n",
       "      <td>3.0150</td>\n",
       "      <td>-0.76</td>\n",
       "      <td>3.0420</td>\n",
       "      <td>3.0090</td>\n",
       "      <td>5,023,780</td>\n",
       "      <td>15,185.24</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BA.SABADELL</td>\n",
       "      <td>0.8510</td>\n",
       "      <td>0.07</td>\n",
       "      <td>0.8560</td>\n",
       "      <td>0.8452</td>\n",
       "      <td>7,015,644</td>\n",
       "      <td>5,970.51</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BANKINTER</td>\n",
       "      <td>5.9320</td>\n",
       "      <td>-0.07</td>\n",
       "      <td>5.9640</td>\n",
       "      <td>5.9100</td>\n",
       "      <td>319,594</td>\n",
       "      <td>1,899.05</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>BBVA</td>\n",
       "      <td>5.0700</td>\n",
       "      <td>-0.67</td>\n",
       "      <td>5.1070</td>\n",
       "      <td>5.0630</td>\n",
       "      <td>1,377,096</td>\n",
       "      <td>7,003.15</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>CAIXABANK</td>\n",
       "      <td>3.4720</td>\n",
       "      <td>0.26</td>\n",
       "      <td>3.4930</td>\n",
       "      <td>3.4350</td>\n",
       "      <td>5,227,626</td>\n",
       "      <td>18,176.22</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>CELLNEX</td>\n",
       "      <td>41.6500</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>41.7900</td>\n",
       "      <td>41.2000</td>\n",
       "      <td>103,162</td>\n",
       "      <td>4,279.20</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>CIE AUTOMOT.</td>\n",
       "      <td>24.6400</td>\n",
       "      <td>0.90</td>\n",
       "      <td>24.7600</td>\n",
       "      <td>24.2800</td>\n",
       "      <td>63,828</td>\n",
       "      <td>1,569.42</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>ENAGAS</td>\n",
       "      <td>22.0200</td>\n",
       "      <td>1.47</td>\n",
       "      <td>22.0600</td>\n",
       "      <td>21.6400</td>\n",
       "      <td>197,521</td>\n",
       "      <td>4,328.12</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>ENDESA</td>\n",
       "      <td>20.8000</td>\n",
       "      <td>0.63</td>\n",
       "      <td>20.8400</td>\n",
       "      <td>20.5300</td>\n",
       "      <td>187,176</td>\n",
       "      <td>3,879.64</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>FERROVIAL</td>\n",
       "      <td>25.0900</td>\n",
       "      <td>0.97</td>\n",
       "      <td>25.2400</td>\n",
       "      <td>24.8500</td>\n",
       "      <td>227,616</td>\n",
       "      <td>5,716.42</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>FLUIDRA</td>\n",
       "      <td>24.8800</td>\n",
       "      <td>-0.88</td>\n",
       "      <td>25.2200</td>\n",
       "      <td>24.7600</td>\n",
       "      <td>110,688</td>\n",
       "      <td>2,759.17</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>GRIFOLS CL.A</td>\n",
       "      <td>18.9250</td>\n",
       "      <td>-0.97</td>\n",
       "      <td>19.1000</td>\n",
       "      <td>18.8350</td>\n",
       "      <td>134,784</td>\n",
       "      <td>2,556.24</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>IAG</td>\n",
       "      <td>1.4540</td>\n",
       "      <td>-1.46</td>\n",
       "      <td>1.4830</td>\n",
       "      <td>1.4500</td>\n",
       "      <td>8,114,181</td>\n",
       "      <td>11,878.24</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>IBERDROLA</td>\n",
       "      <td>10.8600</td>\n",
       "      <td>-0.37</td>\n",
       "      <td>10.9600</td>\n",
       "      <td>10.8150</td>\n",
       "      <td>4,950,247</td>\n",
       "      <td>53,706.68</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>INDITEX</td>\n",
       "      <td>22.0000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>22.0500</td>\n",
       "      <td>21.8000</td>\n",
       "      <td>1,013,878</td>\n",
       "      <td>22,248.44</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>INDRA A</td>\n",
       "      <td>10.6600</td>\n",
       "      <td>-0.56</td>\n",
       "      <td>10.7100</td>\n",
       "      <td>10.6000</td>\n",
       "      <td>132,462</td>\n",
       "      <td>1,412.26</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:21:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>INM.COLONIAL</td>\n",
       "      <td>7.5850</td>\n",
       "      <td>1.47</td>\n",
       "      <td>7.6200</td>\n",
       "      <td>7.4350</td>\n",
       "      <td>208,701</td>\n",
       "      <td>1,575.78</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>MAPFRE</td>\n",
       "      <td>1.7560</td>\n",
       "      <td>0.23</td>\n",
       "      <td>1.7720</td>\n",
       "      <td>1.7530</td>\n",
       "      <td>2,436,838</td>\n",
       "      <td>4,305.47</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:21:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>MELIA HOTELS</td>\n",
       "      <td>7.6200</td>\n",
       "      <td>1.40</td>\n",
       "      <td>7.6700</td>\n",
       "      <td>7.5300</td>\n",
       "      <td>237,370</td>\n",
       "      <td>1,806.83</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>MERLIN</td>\n",
       "      <td>10.5400</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10.5900</td>\n",
       "      <td>10.5200</td>\n",
       "      <td>186,757</td>\n",
       "      <td>1,972.88</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>NATURGY</td>\n",
       "      <td>28.4600</td>\n",
       "      <td>0.42</td>\n",
       "      <td>28.5600</td>\n",
       "      <td>28.0900</td>\n",
       "      <td>47,586</td>\n",
       "      <td>1,352.59</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>PHARMA MAR</td>\n",
       "      <td>69.2800</td>\n",
       "      <td>-1.84</td>\n",
       "      <td>70.4000</td>\n",
       "      <td>69.1600</td>\n",
       "      <td>25,092</td>\n",
       "      <td>1,743.68</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:23:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>R.E.C.</td>\n",
       "      <td>19.1100</td>\n",
       "      <td>0.45</td>\n",
       "      <td>19.1600</td>\n",
       "      <td>18.9550</td>\n",
       "      <td>133,476</td>\n",
       "      <td>2,544.16</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>REPSOL</td>\n",
       "      <td>15.8800</td>\n",
       "      <td>3.02</td>\n",
       "      <td>16.0450</td>\n",
       "      <td>15.3900</td>\n",
       "      <td>2,760,539</td>\n",
       "      <td>43,345.48</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>ROVI</td>\n",
       "      <td>60.0000</td>\n",
       "      <td>-0.99</td>\n",
       "      <td>60.5000</td>\n",
       "      <td>59.8000</td>\n",
       "      <td>27,131</td>\n",
       "      <td>1,631.34</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:18:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>SIEMENS GAME</td>\n",
       "      <td>18.0250</td>\n",
       "      <td>-0.06</td>\n",
       "      <td>18.0500</td>\n",
       "      <td>18.0050</td>\n",
       "      <td>1,549,486</td>\n",
       "      <td>27,936.40</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>SOLARIA</td>\n",
       "      <td>22.7700</td>\n",
       "      <td>2.71</td>\n",
       "      <td>22.8200</td>\n",
       "      <td>22.0900</td>\n",
       "      <td>182,454</td>\n",
       "      <td>4,110.71</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:24:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>TELEFONICA</td>\n",
       "      <td>4.6380</td>\n",
       "      <td>-0.94</td>\n",
       "      <td>4.6750</td>\n",
       "      <td>4.6250</td>\n",
       "      <td>2,346,281</td>\n",
       "      <td>10,892.69</td>\n",
       "      <td>07/06/2022</td>\n",
       "      <td>12:25:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Name      Last % Dif.      High       Low     Volume  \\\n",
       "0        ACCIONA  189.2000  -0.16  190.0000  187.4000     20,137   \n",
       "1       ACERINOX   11.6800  -0.34   11.7600   11.6050    823,017   \n",
       "2            ACS   26.7900  -0.22   26.9100   26.6600    144,000   \n",
       "3           AENA  141.6000  -0.32  142.1000  141.3500     20,318   \n",
       "4       ALMIRALL   10.3900   0.39   10.8200   10.3600    403,502   \n",
       "5        AMADEUS   57.2000  -0.97   57.6400   56.7800     84,981   \n",
       "6    ARCELORMIT.   30.6950  -0.34   30.8150   30.4500     79,819   \n",
       "7    B.SANTANDER    3.0150  -0.76    3.0420    3.0090  5,023,780   \n",
       "8    BA.SABADELL    0.8510   0.07    0.8560    0.8452  7,015,644   \n",
       "9      BANKINTER    5.9320  -0.07    5.9640    5.9100    319,594   \n",
       "10          BBVA    5.0700  -0.67    5.1070    5.0630  1,377,096   \n",
       "11     CAIXABANK    3.4720   0.26    3.4930    3.4350  5,227,626   \n",
       "12       CELLNEX   41.6500  -0.10   41.7900   41.2000    103,162   \n",
       "13  CIE AUTOMOT.   24.6400   0.90   24.7600   24.2800     63,828   \n",
       "14        ENAGAS   22.0200   1.47   22.0600   21.6400    197,521   \n",
       "15        ENDESA   20.8000   0.63   20.8400   20.5300    187,176   \n",
       "16     FERROVIAL   25.0900   0.97   25.2400   24.8500    227,616   \n",
       "17       FLUIDRA   24.8800  -0.88   25.2200   24.7600    110,688   \n",
       "18  GRIFOLS CL.A   18.9250  -0.97   19.1000   18.8350    134,784   \n",
       "19           IAG    1.4540  -1.46    1.4830    1.4500  8,114,181   \n",
       "20     IBERDROLA   10.8600  -0.37   10.9600   10.8150  4,950,247   \n",
       "21       INDITEX   22.0000   0.00   22.0500   21.8000  1,013,878   \n",
       "22       INDRA A   10.6600  -0.56   10.7100   10.6000    132,462   \n",
       "23  INM.COLONIAL    7.5850   1.47    7.6200    7.4350    208,701   \n",
       "24        MAPFRE    1.7560   0.23    1.7720    1.7530  2,436,838   \n",
       "25  MELIA HOTELS    7.6200   1.40    7.6700    7.5300    237,370   \n",
       "26        MERLIN   10.5400   0.00   10.5900   10.5200    186,757   \n",
       "27       NATURGY   28.4600   0.42   28.5600   28.0900     47,586   \n",
       "28    PHARMA MAR   69.2800  -1.84   70.4000   69.1600     25,092   \n",
       "29        R.E.C.   19.1100   0.45   19.1600   18.9550    133,476   \n",
       "30        REPSOL   15.8800   3.02   16.0450   15.3900  2,760,539   \n",
       "31          ROVI   60.0000  -0.99   60.5000   59.8000     27,131   \n",
       "32  SIEMENS GAME   18.0250  -0.06   18.0500   18.0050  1,549,486   \n",
       "33       SOLARIA   22.7700   2.71   22.8200   22.0900    182,454   \n",
       "34    TELEFONICA    4.6380  -0.94    4.6750    4.6250  2,346,281   \n",
       "\n",
       "   Turnover (€ Thousands)        Date      Time  \n",
       "0                3,808.69  07/06/2022  12:25:10  \n",
       "1                9,619.55  07/06/2022  12:25:15  \n",
       "2                3,862.44  07/06/2022  12:24:38  \n",
       "3                2,879.65  07/06/2022  12:24:57  \n",
       "4                4,256.16  07/06/2022  12:21:42  \n",
       "5                4,863.99  07/06/2022  12:24:53  \n",
       "6                2,443.79  07/06/2022  12:25:05  \n",
       "7               15,185.24  07/06/2022  12:25:17  \n",
       "8                5,970.51  07/06/2022  12:25:06  \n",
       "9                1,899.05  07/06/2022  12:25:06  \n",
       "10               7,003.15  07/06/2022  12:25:06  \n",
       "11              18,176.22  07/06/2022  12:25:03  \n",
       "12               4,279.20  07/06/2022  12:25:01  \n",
       "13               1,569.42  07/06/2022  12:23:14  \n",
       "14               4,328.12  07/06/2022  12:24:45  \n",
       "15               3,879.64  07/06/2022  12:25:06  \n",
       "16               5,716.42  07/06/2022  12:25:07  \n",
       "17               2,759.17  07/06/2022  12:25:16  \n",
       "18               2,556.24  07/06/2022  12:25:11  \n",
       "19              11,878.24  07/06/2022  12:25:01  \n",
       "20              53,706.68  07/06/2022  12:25:15  \n",
       "21              22,248.44  07/06/2022  12:25:09  \n",
       "22               1,412.26  07/06/2022  12:21:07  \n",
       "23               1,575.78  07/06/2022  12:24:15  \n",
       "24               4,305.47  07/06/2022  12:21:35  \n",
       "25               1,806.83  07/06/2022  12:23:32  \n",
       "26               1,972.88  07/06/2022  12:23:52  \n",
       "27               1,352.59  07/06/2022  12:23:49  \n",
       "28               1,743.68  07/06/2022  12:23:58  \n",
       "29               2,544.16  07/06/2022  12:24:55  \n",
       "30              43,345.48  07/06/2022  12:24:57  \n",
       "31               1,631.34  07/06/2022  12:18:50  \n",
       "32              27,936.40  07/06/2022  12:25:02  \n",
       "33               4,110.71  07/06/2022  12:24:25  \n",
       "34              10,892.69  07/06/2022  12:25:18  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#rows X_path\n",
    "row_Xpath = '//*[@id=\"ctl00_Contenido_tblAcciones\"]/tbody/tr'\n",
    "\n",
    "#header X_path\n",
    "header_X_path = '//*[@id=\"ctl00_Contenido_tblAcciones\"]/tbody/tr[1]/th'\n",
    "\n",
    "\n",
    "# configure webdriver\n",
    "options = webdriver.FirefoxOptions()\n",
    "options.headless = True  # hide GUI\n",
    "# set window size to native GUI size\n",
    "options.add_argument(\"--window-size=1920,1080\")\n",
    "options.add_argument(\"start-maximized\")\n",
    "\n",
    "# get method to launch the URL\n",
    "driver = webdriver.Firefox(options=options)\n",
    "driver.get(url)\n",
    "\n",
    "\n",
    "# to keep columns names\n",
    "columns_name = []\n",
    "\n",
    "#keep columns names\n",
    "for name in driver.find_elements(by=By.XPATH, value=header_X_path):\n",
    "    columns_name.append(name.text)\n",
    "\n",
    "#define empy dataframe with columns_names\n",
    "df = pd.DataFrame(columns=columns_name)\n",
    "\n",
    "# #keep rows\n",
    "for x in driver.find_elements(by=By.XPATH, value=row_Xpath):\n",
    "    rows = x.find_elements_by_tag_name('td')\n",
    "    data = []\n",
    "    for row in rows:\n",
    "        data.append(row.text)\n",
    "    if len(data) != 0:\n",
    "      df.loc[len(df.index)] = data\n",
    "\n",
    "#to close the browser\n",
    "driver.close()\n",
    "\n",
    "#show the table\n",
    "display(df)\n",
    "\n",
    "#save to cvs\n",
    "df.to_csv('data2.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "214112c9",
   "metadata": {},
   "source": [
    "$~$\n",
    "___\n",
    "####  Exercise 2\n",
    "\n",
    "$~$\n",
    "\n",
    "Document in a word your data set generated with the information that the different Kaggle files have.\n",
    "\n",
    "\n",
    "$~$\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a6760ed",
   "metadata": {},
   "source": [
    "**IBEX 35 prices**.\n",
    "\n",
    "The IBEX 35 (Spanish Stock Market Index) is the reference stock market index of the Madrid Stock Exchange and is made up of 35 companies. This dataset contains the share prices of IBEX35 companies. The data has been collected from The Madrid stock exchange  website on 31/05/2022 at 19:05 p.m.\n",
    "\n",
    "Contents\n",
    "\n",
    "The categories of this dataset:\n",
    "+  Name: Company name.\n",
    "+ Last: Last recorded price\n",
    "+ % Diff.: Difference between the last registered price and the current one in %-\n",
    "+ Maximum: Maximum price reached.\n",
    "+ Minimum: Minimum price reached.\n",
    "+ Volume: Total shares.\n",
    "+ Cash: Total share capital (thousand €):\n",
    "+ Date: Date of registration.\n",
    "\n",
    "Prices expressed in euros. Cash expressed in thousands of euros. The volume and cash for each security includes all operations carried out until the close of the trading session."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7caa1ff",
   "metadata": {},
   "source": [
    "$~$\n",
    "___\n",
    "####  Exercise 3\n",
    "\n",
    "$~$\n",
    "\n",
    "Choose a web page you want and do web scraping using the Scrapy library\n",
    "\n",
    "$~$\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f86b191",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "For this part, we are going to collect the last 195 press articles from the newspaper **El Pais* ([its version in English](https://english.elpais.com/)), specifically from the economy section.\n",
    "\n",
    "$~$\n",
    "\n",
    "Of the articles we are going to save :\n",
    "$~$\n",
    "\n",
    "+   **url** = the link (where you can find the article)\n",
    "+   **date**= the date the article was published\n",
    "+   **author** = name of the journalist who wrote the article\n",
    "+ **art_text** = full text of the article.\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "906952ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ArticleSpider(scrapy.Spider):\n",
    "    name = \"articles\"\n",
    "    \n",
    " \n",
    "    start_urls = ['https://english.elpais.com/economy-and-business/' ]\n",
    "\n",
    "    def parse(self, response):\n",
    "        author_page_links = response.xpath(\"//header/h2/a/@href\")\n",
    "        yield from response.follow_all(author_page_links, self.parse_author)\n",
    "\n",
    "        pagination_links = \"https://english.elpais.com/\" + response.xpath(\"/html/body/div/main/div/div/a/@href\").get()\n",
    "        yield from response.follow_all(pagination_links, self.parse)\n",
    "\n",
    "    def parse_author(self, response):\n",
    "        def extract_with_css(query):\n",
    "            return response.xpath(query).get(default='').strip()\n",
    "        def extract_all(query):\n",
    "            return response.xpath(query).getall()\n",
    "\n",
    "        yield {\n",
    "            'url': response.url,\n",
    "            'date': extract_with_css('//*[@id=\"article_date_p\"]/text()'),\n",
    "            'title': extract_with_css('//header/div/h1/text()'),\n",
    "            'author': extract_with_css('/html/body/div/article/div/div/div/a/text()'),\n",
    "            'text_art': extract_all('/html/body/div/article/div/p/text()'),\n",
    "        }\n",
    "FILE_NAME = 'data.csv'\n",
    "SETTINGS = {\n",
    "            'FEED_FORMAT': 'csv',\n",
    "            'FEED_URI': FILE_NAME,\n",
    "            'DOWNLOAD_DELAY': 1,\n",
    "            } \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8aca506",
   "metadata": {},
   "outputs": [],
   "source": [
    "process = CrawlerProcess(SETTINGS)\n",
    "process.crawl(ArticleSpider) \n",
    "process.start()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bbd0a44c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(297, 5)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=pd.read_csv('data.csv')\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "80eb47ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>tittle</th>\n",
       "      <th>author</th>\n",
       "      <th>text_art</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>https://english.elpais.com/culture/2022-06-01/...</td>\n",
       "      <td>01 Jun 2022</td>\n",
       "      <td>Johnny Depp vs Amber Heard: The form the jury ...</td>\n",
       "      <td>Miguel Jiménez</td>\n",
       "      <td>The contentious , is now in the hands of the j...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>https://english.elpais.com/science-tech/2022-0...</td>\n",
       "      <td>01 Jun 2022</td>\n",
       "      <td>Microbe study sheds light on a critical step i...</td>\n",
       "      <td>Nuño Domínguez</td>\n",
       "      <td>A research study on a microscopic microbe foun...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>https://english.elpais.com/international/2022-...</td>\n",
       "      <td>01 Jun 2022</td>\n",
       "      <td>Colombia’s Trump, Rodolfo Hernández: ‘Ideally,...</td>\n",
       "      <td>Sally Palomino</td>\n",
       "      <td>Rodolfo Hernández cannot hide his sexism. In a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>https://english.elpais.com/international/2022-...</td>\n",
       "      <td>01 Jun 2022</td>\n",
       "      <td>Resistance on rails: How Ukraine’s biggest emp...</td>\n",
       "      <td>Luis de Vega (Enviado Especial)</td>\n",
       "      <td>Ukraine’s national railway company, Ukrzalizni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>218</th>\n",
       "      <td>https://english.elpais.com/international/2022-...</td>\n",
       "      <td>01 Jun 2022</td>\n",
       "      <td>Resistance on rails: How Ukraine’s biggest emp...</td>\n",
       "      <td>Luis de Vega (Enviado Especial)</td>\n",
       "      <td>Ukraine’s national railway company, Ukrzalizni...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   url         date  \\\n",
       "44   https://english.elpais.com/culture/2022-06-01/...  01 Jun 2022   \n",
       "52   https://english.elpais.com/science-tech/2022-0...  01 Jun 2022   \n",
       "53   https://english.elpais.com/international/2022-...  01 Jun 2022   \n",
       "54   https://english.elpais.com/international/2022-...  01 Jun 2022   \n",
       "218  https://english.elpais.com/international/2022-...  01 Jun 2022   \n",
       "\n",
       "                                                tittle  \\\n",
       "44   Johnny Depp vs Amber Heard: The form the jury ...   \n",
       "52   Microbe study sheds light on a critical step i...   \n",
       "53   Colombia’s Trump, Rodolfo Hernández: ‘Ideally,...   \n",
       "54   Resistance on rails: How Ukraine’s biggest emp...   \n",
       "218  Resistance on rails: How Ukraine’s biggest emp...   \n",
       "\n",
       "                              author  \\\n",
       "44                    Miguel Jiménez   \n",
       "52                    Nuño Domínguez   \n",
       "53                    Sally Palomino   \n",
       "54   Luis de Vega (Enviado Especial)   \n",
       "218  Luis de Vega (Enviado Especial)   \n",
       "\n",
       "                                              text_art  \n",
       "44   The contentious , is now in the hands of the j...  \n",
       "52   A research study on a microscopic microbe foun...  \n",
       "53   Rodolfo Hernández cannot hide his sexism. In a...  \n",
       "54   Ukraine’s national railway company, Ukrzalizni...  \n",
       "218  Ukraine’s national railway company, Ukrzalizni...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=\"date\").head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e4504b",
   "metadata": {},
   "source": [
    "$~$\n",
    "\n",
    "+ we print an article to see the result of the extraction:\n",
    "\n",
    "$~$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Johnny Depp vs Amber Heard: The form the jury will use to reach its verdict\n",
      "----------\n",
      "The contentious , is now in the hands of the jury. The seven-member panel heard the closing arguments on May 27, and met to deliberate for the first time. Since then, it has concluded two days of deliberations without reaching a decision. The jury will resume deliberations on Wednesday morning, but no one knows how long it will take the jury to reach a verdict.,As the end of the trial draws near, people have been waiting in huge lines and even camping overnight outside the courthouse to secure a seat in one of the most-talked-about cases in recent history. It is not a criminal case; the jury will not rule on the allegations of abuse and assault that have been raised in the trial. Instead, they will focus on the subject of the lawsuit: defamation.,Depp is suing Heard for $50 million for defaming him in an ,in 2018, in which she described herself as “a public figure representing domestic abuse.” Although Depp is not mentioned by name, he argues that it was obvious who she was referring to. Heard responded to Depp’s complaint by countersuing him for $100 million, claiming that his lawyer smeared her by calling her a liar. While Heard maintains that Depp , and even sexually assaulted her with a bottle, the actor says he , and was, in fact, the victim in the relationship.,To help the jury reach a verdict, the judge has provided the jurors with a special form, ,. The form does not include the words “guilty” or “innocent,” since the jury has not been asked to judge a crime. They must decide whether either the actor or actress has been defamed by the other.,If they rule that Heard defamed Depp and/or that Depp defamed Heard, they must decide on how much compensatory and punitive damages to award.,With respect to Depp’s claim, the jury must decide if three specific statements in Heard’s ,op-ed were defamatory:,- “I spoke up against sexual violence – and faced our culture’s wrath. That has to change.”,- “Then two years ago, I became a public figure representing domestic abuse, and I felt the full force of our culture’s wrath for women who speak out.”,- “I had the rare vantage point of seeing, in real time, how institutions protect men accused of abuse.”,With respect to Heard’s claim, the lawsuit focuses on statements made by Depp’s former lawyer Adam Waldman in ,Although the comments were not made by the actor, Heard argues that he said them on his behalf. The actress’s lawyer also made clear that she is not seeking $100 million in compensation, and only set that amount to send a message to her ex-husband.,In her case, the jury is asked if the following statements were defamatory:,- “Amber Heard and her friends in the media use fake sexual violence allegations as both a sword and a shield, depending on their needs. They have selected some of her sexual violence hoax ‘facts’ as the sword, inflicting them on the public and Mr. Depp.”,- “Quite simply this was an ambush, a hoax. They set Mr. Depp up by calling the cops but the first attempt didn’t do the trick. The officers came to the penthouses, thoroughly searched and interviewed, and left after seeing no damage to face or property. So Amber and her friends spilled a little wine and roughed the place up, got their stories straight under the direction of a lawyer and publicists, and then placed a second call to 911.”,- ”[W]e have reached the beginning of the end of Ms. Heard’s abuse hoax against Johnny Depp.”,For each of the six statements, the jurors are asked a series of “yes or no” questions to determine whether all the elements of defamation have been proven, such as “The statement was false” and “The defamation implication was designed and intended.” If “yes” has been answered to all questions, the jury must set an amount for compensatory and punitive damages.,On the final day of the trial, Depp’s lawyers asked the jury to assess Heard’s credibility, arguing the actress was the real abuser in the relationship. Meanwhile Heard’s lawyers claimed her statements were protected by freedom of expression, which is enshrined in the First Amendment of the United States Constitution. They argued that even if the jury did not believe the actress was a victim of,, if they found evidence of psychological or financial abuse, she would be protected by freedom of expression. The case is now in the hands of the jury.,The ruling, however, can be appealed. The case is being heard in Fairfax County District court in Virginia. This location has been chosen because , is printed there and because the newspaper also has its digital servers in the town. But some lawyers have questioned Fairfax’s jurisdiction, given that both Heard and Depp are based in California. While it does not seem likely, the possibility of a new Depp vs Heard trial being heard in the next few years cannot be ruled out.\n"
     ]
    }
   ],
   "source": [
    "print(df[\"tittle\"][44])\n",
    "print(\"-\"*10)\n",
    "print(df[\"text_art\"][44])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dca1eb0c",
   "metadata": {},
   "source": [
    "$~$\n",
    "___\n",
    "####  Conclusions\n",
    "\n",
    "$~$\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "___\n",
    "$~$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bed5aa6a",
   "metadata": {},
   "source": [
    "+ There are different libraries that we use to scrape websites. Here, we have reviewed 3 of the most common ones. (**BeautifulSoup, Selenium, Scrapy**)\n",
    "\n",
    "+ For the use of these libraries they require a minimum previous knowledge of HTML\n",
    "\n",
    "+ We have achieved the same result using **Beautifulsoup** and **Selenium**, however, the **Beautifulsoup** library has been more efficient since its handling (and learning curve) make it a more user-friendly tool.\n",
    "\n",
    "+ The articles that we have obtained could serve us as data for any **NLP** project"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0eedf5c",
   "metadata": {},
   "source": [
    "___\n",
    "$~$\n",
    "\n",
    "####  *References*:\n",
    "\n",
    "+ [Beautiful Soup: Build a Web Scraper With Python](https://realpython.com/beautiful-soup-web-scraper-python/) \n",
    "+ [Selenium Example Explained](https://selenium-python.readthedocs.io/getting-started.html#simple-usage) \n",
    "+ [Learn Scrapy](https://www.tutorialspoint.com/scrapy/index.htm) \n",
    "\n",
    "$~$\n",
    "___"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c196e456b92eeb3a60b06f99b3c60ac6c4763f8bb1169beee1475ccb9dd5a29f"
  },
  "kernelspec": {
   "display_name": "Python 3.9.5 64-bit ('3.9.5')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}