How_to_Use_Mindat_API.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "2bVXRaAPa5HK",
        "N-obNLZGa5HL",
        "lCu8IZeLa5HM",
        "GNcVjmUia5HM",
        "tJywJN-ia5HN",
        "Q3EBzJhIa5HN",
        "NuLEEpYSa5HN",
        "bzPZb-V_kCfe"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/ChuBL/How-to-Use-Mindat-API/blob/main/How_to_Use_Mindat_API.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ],
      "metadata": {
        "id": "qrMhTOpOyjAP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# How to Use OpenMindat Data API to Query and Download Datasets\n"
      ],
      "metadata": {
        "id": "u65PipxXuNbO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 0. Access Your Mindat API Token"
      ],
      "metadata": {
        "id": "OOT8B2v7odok"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "[How to Get My Mindat API Key or Token?](https://www.mindat.org/a/how_to_get_my_mindat_api_key)"
      ],
      "metadata": {
        "id": "59gsuIxcoiIU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Dependencies (Please run this section first)"
      ],
      "metadata": {
        "id": "YmvOE3ClX207"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pathlib import Path\n",
        "import os\n",
        "import sys\n",
        "import json\n",
        "import re\n",
        "import pprint\n",
        "import requests"
      ],
      "metadata": {
        "id": "gpxJUYWHaPCw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# You can change this working path according to your preference\n",
        "# If the folder didn't show up in the left column, click the refresh button\n",
        "WORKING_DIR = \"/content/MindatAPI_folder/\"\n",
        "Path(WORKING_DIR).mkdir(parents=True, exist_ok=True)"
      ],
      "metadata": {
        "id": "zyImDhp2aYNq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You should **avoid** placing your API token explicitly in your code if you plan to share it.\n",
        "\n",
        "For example, you can drag a text file of your token to the working directory to upload it, then access it using the following code. Or you can input it manually."
      ],
      "metadata": {
        "id": "RhAUoA9pcIQv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You should find your API key at [mindat.org](https://www.mindat.org/) >> `My Home Page` >> `Edit My Page`, at the bottom of the page.\n"
      ],
      "metadata": {
        "id": "nT7M_hAccPWR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "YOUR_API_KEY = \"\""
      ],
      "metadata": {
        "id": "QSoVg4g-eQw0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%script false --no-raise-error\n",
        "# comment out the first line to activate this code block\n",
        "YOUR_API_KEY = input()"
      ],
      "metadata": {
        "id": "39HPkjIKaMBo"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    assert 0 != len(YOUR_API_KEY)\n",
        "except AssertionError:\n",
        "    raise Exception(\"Please set a valid API token before the start!\")"
      ],
      "metadata": {
        "id": "tYg14BcheNlO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Use Cases\n"
      ],
      "metadata": {
        "id": "BZb3kkEaa5HJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the Geomaterial Items with Selected Fields\n"
      ],
      "metadata": {
        "id": "2bVXRaAPa5HK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Examples for Mindat API fields: `fields=id,longid,guid,name,updttime,mindat_formula,mindat_formula_note,ima_formula,ima_status,...`\n",
        "\n",
        "For entire fields list, see: [OpenMindat API Documentation](https://api.mindat.org/schema/redoc/#tag/items/operation/items_list)"
      ],
      "metadata": {
        "id": "Re-0qXaqa5HK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "A comprehensive field explanations document can be found [here](https://github.com/smrgeoinfo/How-to-Use-Mindat-API/blob/main/geomaterialfields.csv)."
      ],
      "metadata": {
        "id": "XDNc6JvSa5HL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "select_file_name = \"mindat_items_select.json\"\n",
        "select_file_path = Path(WORKING_DIR, select_file_name)\n",
        "select_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c57c5978-5000-434c-ebb7-4fd8ef54169f",
        "id": "bbbi4j5Pa5HL"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/MindatAPI_folder/mindat_items_select.json')"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# set your selected fields here\n",
        "fields_str = 'id,name,mindat_formula'\n",
        "\n",
        "with open(select_file_path, 'w') as f:\n",
        "    params = {\n",
        "        'fields': fields_str, # put your selected fields here\n",
        "        'format': 'json'\n",
        "    }\n",
        "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
        "                    params=params,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()[\"results\"]\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    # Uncomment the following code to obtain more than ten results\n",
        "    # while True:\n",
        "    #     try:\n",
        "    #         next_url = response.json()[\"next\"]\n",
        "    #         response = requests.get(next_url, headers=headers)\n",
        "    #         json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "    #     except requests.exceptions.MissingSchema as e:\n",
        "    #         # This error indicates the `next_url` is none\n",
        "    #         break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data['results'])) + \" entries to \" + str(select_file_path))"
      ],
      "metadata": {
        "id": "sjJWzJxba5HL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the Geomaterial Items with Omitted Fields"
      ],
      "metadata": {
        "id": "N-obNLZGa5HL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "omit_file_name = \"mindat_items_omit.json\"\n",
        "omit_file_path = Path(WORKING_DIR, omit_file_name)\n",
        "omit_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31f39b15-4f82-43df-a540-35aa45a2901f",
        "id": "EkgONGY5a5HM"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/MindatAPI_folder/mindat_items_omit.json')"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "omit_str = 'id,name,updttime'\n",
        "\n",
        "\n",
        "with open(omit_file_path, 'w') as f:\n",
        "    params = {\n",
        "        'omit': omit_str,\n",
        "        'format': 'json'\n",
        "    }\n",
        "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
        "                    params=params,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()[\"results\"]\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    # Uncomment the following code to obtain more than ten results\n",
        "    # while True:\n",
        "    #     try:\n",
        "    #         next_url = response.json()[\"next\"]\n",
        "    #         response = requests.get(next_url, headers=headers)\n",
        "    #         json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "    #     except requests.exceptions.MissingSchema as e:\n",
        "    #         # This error indicates the `next_url` is none\n",
        "    #         break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data['results'])) + \" entries to \" + str(omit_file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9d527b84-a783-4e0e-da97-dd0010f4e510",
        "id": "u-E86IVna5HM"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved 10 entries to /content/MindatAPI_folder/mindat_items_omit.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the IMA-Approved Mineral Geomaterial Items"
      ],
      "metadata": {
        "id": "lCu8IZeLa5HM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "ima_file_name = \"mindat_items_IMA.json\"\n",
        "ima_file_path = Path(WORKING_DIR, ima_file_name)\n",
        "ima_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "514710ff-5eeb-4db2-eb59-99e39759d58d",
        "id": "c71OrsJia5HM"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/MindatAPI_folder/mindat_items_IMA.json')"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This query will obtain thousands of mineral species and may take a couple of minutes(4min ~)."
      ],
      "metadata": {
        "id": "aliXJVnXHjWr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open(ima_file_path, 'w') as f:\n",
        "    params = {\n",
        "        'format': 'json'\n",
        "    }\n",
        "    response = requests.get(MINDAT_API_URL+\"/minerals_ima/\",\n",
        "                    params=params,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()[\"results\"]\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    while True:\n",
        "        try:\n",
        "            next_url = response.json()[\"next\"]\n",
        "            response = requests.get(next_url, headers=headers)\n",
        "            json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "        except requests.exceptions.MissingSchema as e:\n",
        "            # This error indicates the `next_url` is none\n",
        "            break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data['results'])) + \" entries to \" + str(ima_file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "12ad75fe-3822-49c9-d974-8f85a8dae35b",
        "id": "hzl4DWrza5HM"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved 5943 entries to /content/MindatAPI_folder/mindat_items_IMA.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the Geomaterial Items with Filters"
      ],
      "metadata": {
        "id": "GNcVjmUia5HM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "filter_file_name = \"mindat_items_filter.json\"\n",
        "filter_file_path = Path(WORKING_DIR, filter_file_name)\n",
        "filter_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0bc5e502-9d42-4a01-c5bc-962fbfa523bb",
        "id": "PDMxIxUca5HM"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/MindatAPI_folder/mindat_items_filter.json')"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This query will obtain full-sized IMA mineral information from Mindat and may take a couple of minutes(6min ~)."
      ],
      "metadata": {
        "id": "vLn5hXkQTwQS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "filter_dict = {\n",
        "          \"ima_status\": [\n",
        "                    \"APPROVED\"\n",
        "                ], # an alternative way to obtain the IMA-approved minerals\n",
        "          #\"id_in\": '1,2',\n",
        "          #\"colour\": \"green\",\n",
        "          'format': 'json',\n",
        "          }\n",
        "\n",
        "with open(filter_file_path, 'w') as f:\n",
        "\n",
        "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
        "                    params=filter_dict,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()[\"results\"]\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    while True:\n",
        "        try:\n",
        "            next_url = response.json()[\"next\"]\n",
        "            response = requests.get(next_url, headers=headers)\n",
        "            json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "        except requests.exceptions.MissingSchema as e:\n",
        "            # This error indicates the `next_url` is none\n",
        "            break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data[\"results\"])) + \" entries to \" + str(filter_file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2b2a8b7c-0c24-4a41-f626-1daf91aaf07e",
        "id": "08dR3vGVa5HN"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved 5943 entries to /content/MindatAPI_folder/mindat_items_filter.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the Geomaterial Items with Fuzzy Search"
      ],
      "metadata": {
        "id": "tJywJN-ia5HN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "file_name = \"mindat_items_search.json\"\n",
        "file_path = Path(WORKING_DIR, file_name)\n",
        "file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0876fe9a-fe5c-4099-be6a-bd7c936fcdf1",
        "id": "T4BI-fZ3a5HN"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/drive/MyDrive/MindatAPI_folder/mindat_items_search.json')"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "filter_dict = {'q': 'quartz and ruby', # input what you about to search same as the home page\n",
        "          'format': 'json',\n",
        "          }\n",
        "\n",
        "with open(file_path, 'w') as f:\n",
        "    params = filter_dict\n",
        "\n",
        "    response = requests.get(MINDAT_API_URL+\"/geomaterials_search/\",\n",
        "                    params=params,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data[\"results\"])) + \" entries to \" + str(file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0198b9a4-f436-4ee7-e7a8-53a95ebd869b",
        "id": "fDL5rkR9a5HN"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved 197 entries to /content/drive/MyDrive/MindatAPI_folder/mindat_items_search.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get Single Locality"
      ],
      "metadata": {
        "id": "Q3EBzJhIa5HN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "sig_loc_file_name = \"mindat_loc_single.json\"\n",
        "sig_loc_file_path = Path(WORKING_DIR, sig_loc_file_name)\n",
        "sig_loc_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8c8738d-93c0-4e48-9cc5-10e0256da7e4",
        "id": "mNm2Tftka5HN"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/drive/MyDrive/MindatAPI_folder/mindat_loc_single.json')"
            ]
          },
          "metadata": {},
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with open(sig_loc_file_path, 'w') as f:\n",
        "    # The Mindat locality page for Moscow, Idaho\n",
        "    # https://www.mindat.org/loc-58150.html\n",
        "    locality_id = '306797'\n",
        "    response = requests.get(MINDAT_API_URL + '/localities/' + locality_id,\n",
        "                    headers=headers)\n",
        "    result_data = response.json()\n",
        "    json_data = {\"results\": [result_data]}\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved the locality to \" + str(sig_loc_file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "17b1813f-ab2a-4057-a005-d3fc97fe982a",
        "id": "Fb-mwTn-a5HN"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved the locality to /content/drive/MyDrive/MindatAPI_folder/mindat_loc_single.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get Locality List with multiple IDs"
      ],
      "metadata": {
        "id": "NuLEEpYSa5HN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "loc_file_name = \"mindat_loc_list.json\"\n",
        "loc_file_path = Path(WORKING_DIR, loc_file_name)\n",
        "loc_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d0ccf98a-1202-4b36-be4c-48e63ac572ee",
        "id": "E7Vf57s-a5HO"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/drive/MyDrive/MindatAPI_folder/mindat_loc_list.json')"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "with open(loc_file_path, 'w') as f:\n",
        "    params = {\n",
        "        # Mindat loc ids for Moscow and Los Angeles\n",
        "        'id__in': '58150,3493',\n",
        "        'format': 'json'\n",
        "    }\n",
        "\n",
        "    response = requests.get(MINDAT_API_URL + '/localities/',\n",
        "                    params=params,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()[\"results\"]\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    while True:\n",
        "        try:\n",
        "            next_url = response.json()[\"next\"]\n",
        "            response = requests.get(next_url, headers=headers)\n",
        "            json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "        except requests.exceptions.MissingSchema as e:\n",
        "            # This error indicates the `next_url` is none\n",
        "            break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data['results'])) + \" entries to \" + str(loc_file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d53be60a-a238-47fe-d253-b29501d2a303",
        "id": "yeBGKa7ha5HO"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved 2 entries to /content/drive/MyDrive/MindatAPI_folder/mindat_loc_list.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3. Test Zone"
      ],
      "metadata": {
        "id": "1hZW1eb0KL5f"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get All the Geomaterial Items\n"
      ],
      "metadata": {
        "id": "bzPZb-V_kCfe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "❗Please note that this section of codes will retrieve hundreds of **MB** of data from the Mindat server.\n",
        "\n",
        "❗This operation may consume a significant time and API access quota."
      ],
      "metadata": {
        "id": "dPy7JQpYAbS1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "all_file_name = \"mindat_items_all.json\"\n",
        "all_file_path = Path(WORKING_DIR, all_file_name)\n",
        "all_file_path"
      ],
      "metadata": {
        "id": "JRt7_VwxkbPf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open(all_file_path, 'w') as f:\n",
        "    params = {\n",
        "        'format': 'json',\n",
        "    }\n",
        "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
        "                            params=params,\n",
        "                            headers=headers)\n",
        "    result_data = response.json()[\"results\"]\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    while True:\n",
        "        try:\n",
        "            next_url = response.json()[\"next\"]\n",
        "            response = requests.get(next_url, headers=headers)\n",
        "            json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "        except requests.exceptions.MissingSchema as e:\n",
        "            # This error indicates the `next_url` is none\n",
        "            break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data['results'])) + \" entries to \" + str(all_file_path))"
      ],
      "metadata": {
        "id": "VzuHQxA4kx4H"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the Geomaterial Items with Filters"
      ],
      "metadata": {
        "id": "N1KNupx6DQ9T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MINDAT_API_URL = \"https://api.mindat.org\"\n",
        "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
        "\n",
        "filter_file_name = \"mindat_items_filter.json\"\n",
        "filter_file_path = Path(WORKING_DIR, filter_file_name)\n",
        "filter_file_path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0171f0a9-7c83-4118-c3eb-c7977f520dc9",
        "id": "WIwAxXhWDQ9T"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "PosixPath('/content/drive/MyDrive/MindatAPI_folder/mindat_items_filter.json')"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "filter_dict = {\n",
        "          #\"longid\": \"1:1:13:6\",\n",
        "          #\"colour\": \"green\",\n",
        "          \"guid\": \"c339183f-cf96-4ccc-96c9-6eeb7a0c81d0\",\n",
        "          'format': 'json',\n",
        "          }\n",
        "\n",
        "with open(filter_file_path, 'w') as f:\n",
        "\n",
        "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
        "                    params=filter_dict,\n",
        "                    headers=headers)\n",
        "\n",
        "    result_data = response.json()['results']\n",
        "    json_data = {\"results\": result_data}\n",
        "\n",
        "    # Uncomment the following code to obtain more than ten results\n",
        "    # while True:\n",
        "    #     try:\n",
        "    #         next_url = response.json()[\"next\"]\n",
        "    #         response = requests.get(next_url, headers=headers)\n",
        "    #         json_data[\"results\"] += response.json()['results']\n",
        "\n",
        "    #     except requests.exceptions.MissingSchema as e:\n",
        "    #         # This error indicates the `next_url` is none\n",
        "    #         break\n",
        "\n",
        "    json.dump(json_data, f, indent=4)\n",
        "print(\"Successfully saved \" + str(len(json_data[\"results\"])) + \" entries to \" + str(filter_file_path))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fbd55900-378e-4ea2-a890-4c5cd00e4014",
        "id": "zTV02q_vDQ9T"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully saved 10 entries to /content/drive/MyDrive/MindatAPI_folder/mindat_items_filter.json\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "936-etUwDW-d"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}