01下载数据.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 下载嫦娥公开科学数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting requests\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl (61 kB)\n",
      "\u001b[K     |████████████████████████████████| 61 kB 2.5 MB/s eta 0:00:011\n",
      "\u001b[?25hCollecting chardet<5,>=3.0.2\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/c7/fa589626997dd07bd87d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl (178 kB)\n",
      "\u001b[K     |████████████████████████████████| 178 kB 2.1 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: certifi>=2017.4.17 in /home/moose/anaconda3/envs/pyqt/lib/python3.7/site-packages (from requests) (2020.12.5)\n",
      "Collecting idna<3,>=2.5\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl (58 kB)\n",
      "\u001b[K     |████████████████████████████████| 58 kB 1.7 MB/s eta 0:00:011\n",
      "\u001b[?25hCollecting urllib3<1.27,>=1.21.1\n",
      "  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/09/c6/d3e3abe5b4f4f16cf0dfc9240ab7ce10c2baa0e268989a4e3ec19e90c84e/urllib3-1.26.4-py2.py3-none-any.whl (153 kB)\n",
      "\u001b[K     |████████████████████████████████| 153 kB 1.2 MB/s eta 0:00:01\n",
      "\u001b[?25hInstalling collected packages: urllib3, idna, chardet, requests\n",
      "Successfully installed chardet-4.0.0 idna-2.10 requests-2.25.1 urllib3-1.26.4\n"
     ]
    }
   ],
   "source": [
    "# 依赖包\n",
    "!pip install requests -i https://pypi.tuna.tsinghua.edu.cn/simple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 测试网络爬虫\n",
    "\n",
    "进入[探月工程数据发布与信息服务系统](http://moon.bao.ac.cn/)进行查询，F12找到`proSearch!proSearchList.search`的`GET`请求，在`headers`页面将`GET`部分`copy All`。\n",
    "\n",
    "以下代码中`GET_copyall_from_firefox`就是复制过来的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    GET={\n",
      "        scheme=https\n",
      "        host=moon.bao.ac.cn\n",
      "        filename=/proSearch!proSearchList.search\n",
      "        query={\n",
      "            _dc=1615632080475\n",
      "            pageNo=2\n",
      "            pageSize=10\n",
      "            inputJsonString={\"globalLang\":null,\"DM_DTYPE\":\"SCI\",\"DM_TASKID\":\"CE4\",\"DM_ZHNAME\":\"LCAM\",\"DM_GRADE\":\"2A\",\"DM_CBTIMEStart\":\"2019-01-03 00:00:00\",\"DM_CBTIMEend\":\"2019-01-03 23:59:59\",\"order_by\":\"desc\",\"qsList\":\"\",\"qsStart\":\"\",\"qsEnd\":\"\",\"bianhao\":\"\"}\n",
      "            page=2\n",
      "            start=10\n",
      "            limit=10\n",
      "        }\n",
      "        remote={\n",
      "            Address=159.226.88.59:443\n",
      "        }\n",
      "    }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import atool\n",
    "\n",
    "GET_copyall_from_firefox = {\"GET\":{\"scheme\":\"https\",\"host\":\"moon.bao.ac.cn\",\"filename\":\"/proSearch!proSearchList.search\",\"query\":{\"_dc\":\"1615632080475\",\"pageNo\":\"2\",\"pageSize\":\"10\",\"inputJsonString\":\"{\\\"globalLang\\\":null,\\\"DM_DTYPE\\\":\\\"SCI\\\",\\\"DM_TASKID\\\":\\\"CE4\\\",\\\"DM_ZHNAME\\\":\\\"LCAM\\\",\\\"DM_GRADE\\\":\\\"2A\\\",\\\"DM_CBTIMEStart\\\":\\\"2019-01-03 00:00:00\\\",\\\"DM_CBTIMEend\\\":\\\"2019-01-03 23:59:59\\\",\\\"order_by\\\":\\\"desc\\\",\\\"qsList\\\":\\\"\\\",\\\"qsStart\\\":\\\"\\\",\\\"qsEnd\\\":\\\"\\\",\\\"bianhao\\\":\\\"\\\"}\",\"page\":\"2\",\"start\":\"10\",\"limit\":\"10\"},\"remote\":{\"Address\":\"159.226.88.59:443\"}}}\n",
    "\n",
    "print(atool.pretty(GET_copyall_from_firefox))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将`GET_copyall_from_firefox`中的`scheme`，`host`，`filename`三个字段合成`url`传入`requests.get()`；将`query`作为`param`传入`requests.get()`。\n",
    "\n",
    "特别需要注意的是，`query`中的`inputJsonString`是查询参数，该更改的参数包括任务、载荷、数据类型、数据级别、起止时间等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://moon.bao.ac.cn/proSearch!proSearchList.search\n",
      "{\n",
      "    _dc=1615632080475\n",
      "    pageNo=2\n",
      "    pageSize=10\n",
      "    inputJsonString={\"globalLang\":null,\"DM_DTYPE\":\"SCI\",\"DM_TASKID\":\"CE4\",\"DM_ZHNAME\":\"LCAM\",\"DM_GRADE\":\"2A\",\"DM_CBTIMEStart\":\"2019-01-03 00:00:00\",\"DM_CBTIMEend\":\"2019-01-03 23:59:59\",\"order_by\":\"desc\",\"qsList\":\"\",\"qsStart\":\"\",\"qsEnd\":\"\",\"bianhao\":\"\"}\n",
      "    page=2\n",
      "    start=10\n",
      "    limit=10\n",
      "}\n",
      "\n",
      "<Response [200]>\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "url = GET_copyall_from_firefox['GET']['scheme']+'://'+GET_copyall_from_firefox['GET']['host']+GET_copyall_from_firefox['GET']['filename']\n",
    "param = GET_copyall_from_firefox['GET']['query']\n",
    "\n",
    "print(url)\n",
    "print(atool.pretty(param))\n",
    "\n",
    "r = requests.get(url, param)\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "返回两部分数据：\n",
    "\n",
    "- `returndata`包含了本页查询的条目的所有信息，其中有保存路径，可以通过保存路径获得下载地址。\n",
    "- `totalCount`本次查询命中的全部条目的数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "数据总条目： 10882\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=1048576\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=190330080557186\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=3776\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=1903300805571860\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=1048576\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=190330080547416\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=3776\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=1903300805474160\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=1048576\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=190330080537661\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=3776\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=1903300805376610\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=1048576\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=190330080528387\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=3776\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=1903300805283870\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=1048576\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=1.00(MB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=190330080518553\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n",
      "{\n",
      "    dmTtype=SCI\n",
      "    dmSize=3776\n",
      "    image=images/searchOrder/feiji/defaultPic.png.jpg\n",
      "    paragraph=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmGrade=2A\n",
      "    dmQs=1\n",
      "    imagePath=CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.jpg\n",
      "    author=GRAS\n",
      "    dmTaskid=CE4\n",
      "    title=CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    hoverValues=[\n",
      "        {\n",
      "            value=CE4\n",
      "            key=探测任务\n",
      "        }\n",
      "        {\n",
      "            value=LCAM-1\n",
      "            key=有效载荷\n",
      "        }\n",
      "        {\n",
      "            value=2A\n",
      "            key=级别\n",
      "        }\n",
      "        {\n",
      "            value=3.69(KB)\n",
      "            key=大小\n",
      "        }\n",
      "        {\n",
      "            value=PDS\n",
      "            key=数据格式\n",
      "        }\n",
      "        {\n",
      "            value=1\n",
      "            key=轨道圈数\n",
      "        }\n",
      "        {\n",
      "            value=A\n",
      "            key=版本\n",
      "        }\n",
      "\n",
      "    ]\n",
      "    dmStorepath=/PUBDATA/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "    dmVer=A\n",
      "    objId=1\n",
      "    dmZh=LCAM-1\n",
      "    startTime=2019-01-03 02:29:13.0\n",
      "    dmStorestate=1\n",
      "    id=1903300805185530\n",
      "    endTime=2019-01-03 02:29:13.0\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#print(atool.pretty(r.json()))      # 打印所有数据\n",
    "print('数据总条目：', r.json()['totalCount'])\n",
    "for data in r.json()['returndata']:\n",
    "    print(atool.pretty(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由上可知，每个data数据条目中，`dmStorepath`即为数据文件真正的保存地址。\n",
    "\n",
    "将`/PUBDATA/`替换为`http://moon.bao.ac.cn/cedownload/`即可获得真正的下载链接。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "数据总条目： 10882\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5350_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5351_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5352_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5353_SCI_N_20190103022913_20190103022913_0001_A.2AL\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.2A\n",
      "http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/LCAM/1/2A/2019-03/CE4_GRAS_LCAM-1-5354_SCI_N_20190103022913_20190103022913_0001_A.2AL\n"
     ]
    }
   ],
   "source": [
    "print('数据总条目：', r.json()['totalCount'])\n",
    "for data in r.json()['returndata']:\n",
    "    # 真实的下载链接\n",
    "    print(data['dmStorepath'].replace('/PUBDATA/', 'http://moon.bao.ac.cn/cedownload/'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 设计`get_ChangE_download_link()`函数\n",
    "\n",
    "通过自定义的查询获得嫦娥公开科学数据的下载链接。\n",
    "\n",
    "根据上面的内容，可以设计一个网络爬虫程序，根据给定的`页码`、`页面大小`、`任务`、`数据类型`、`载荷类型`、`数据级别`、`开始时间`、`结束时间`向服务器发起查询，并根据查询结果组装真正的下载链接。\n",
    "\n",
    "目前支持嫦娥1～嫦娥5的数据查询。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import time\n",
    "import atool\n",
    "\n",
    "def get_ChangE_download_link(pageNo, pageSize, task_id='CE4', data_type='SCI', payload='PCAM', data_grade='2B', date_start='default', date_end='today'):\n",
    "    # 默认情况下，根据任务设置查询的起始时间\n",
    "    if date_start == 'default':\n",
    "        if task_id == 'CE5':\n",
    "            date_start = '2020-12-01 00:00:00'\n",
    "        if task_id == 'CE4':\n",
    "            date_start = '2019-01-02 00:00:00'\n",
    "        if task_id == 'CE3':\n",
    "            date_start = '2009-12-31 00:00:00'\n",
    "        if task_id == 'CE2':\n",
    "            date_start = '2010-10-01 00:00:00'\n",
    "        if task_id == 'CE1':\n",
    "            date_start = '2007-10-24 00:00:00'\n",
    "\n",
    "    # 默认情况下，设置查询的结束时间为当天\n",
    "    if date_end == 'today':\n",
    "        date_end = time.strftime(\"%Y-%m-%d %H:%M:%S\", time.localtime())\n",
    "\n",
    "    # 组装参数\n",
    "    url = 'https://' + 'moon.bao.ac.cn' + '/proSearch!proSearchList.search'\n",
    "    inputJsonString = '{\"globalLang\":null,\"DM_DTYPE\":\"%s\",\"DM_TASKID\":\"%s\",\"DM_ZHNAME\":\"%s\",\"DM_GRADE\":\"%s\",\"DM_CBTIMEStart\":\"%s\",\"DM_CBTIMEend\":\"%s\",\"order_by\":\"desc\",\"qsList\":\"\",\"qsStart\":\"\",\"qsEnd\":\"\",\"bianhao\":\"\"}' % (data_type, task_id, payload, data_grade, date_start, date_end)\n",
    "    param = {\n",
    "        '_dc': '1615623500055', \n",
    "        'pageNo': str(pageNo), \n",
    "        'pageSize': str(pageSize), \n",
    "        'inputJsonString': inputJsonString, \n",
    "        'page': str(pageNo), \n",
    "        'start': str((pageNo-1)*pageSize), \n",
    "        'limit': str(pageSize)}\n",
    "        \n",
    "    # 访问网站，查询\n",
    "    r = requests.get(url, param)\n",
    "    if r.ok:\n",
    "        link = []\n",
    "        link.append(r.json()['totalCount'])\n",
    "        for data in r.json()['returndata']:\n",
    "            # print(atool.pretty(data))\n",
    "            # 真实的下载链接\n",
    "            link.append(data['dmStorepath'].replace('/PUBDATA/', 'http://moon.bao.ac.cn/cedownload/'))\n",
    "\n",
    "        return link\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是几个使用的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    7268\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2B\n",
      "\n",
      "]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 默认查询嫦娥4的数据，默认载荷是PCAM，默认数据级别2B\n",
    "links = get_ChangE_download_link(1, 5)\n",
    "if links != None:\n",
    "    print(atool.pretty(links))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    38843\n",
      "    http://moon.bao.ac.cn/cedownload/HPD/2C/CE2_BMYK_HPD_SCI_P_20110522233204_20110523012536_2771_A.2C\n",
      "    http://moon.bao.ac.cn/cedownload/HPD/2C/CE2_BMYK_HPD_SCI_P_20110522213831_20110522233203_2770_A.2C\n",
      "    http://moon.bao.ac.cn/cedownload/HPD/2C/CE2_BMYK_HPD_SCI_P_20110522194457_20110522213830_2769_A.2C\n",
      "    http://moon.bao.ac.cn/cedownload/HPD/2C/CE2_BMYK_HPD_SCI_P_20110522175124_20110522194456_2768_A.2C\n",
      "    http://moon.bao.ac.cn/cedownload/HPD/2C/CE2_BMYK_HPD_SCI_P_20110522155751_20110522175123_2767_A.2C\n",
      "\n",
      "]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 指定任务为嫦娥2，载荷留空查所有，数据级别留空查所有\n",
    "links = get_ChangE_download_link(1, 5, task_id='CE2', payload='', data_grade='')\n",
    "if links != None:\n",
    "    print(atool.pretty(links))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    7349\n",
      "    http://moon.bao.ac.cn/cedownload/CE5ROLL/PCAML/PCAML-I-008/2B/202012/CE5-L_GRAS_PCAML-I-008_SCI_N_20201203054903_20201203054903_0004_A.2B\n",
      "    http://moon.bao.ac.cn/cedownload/PUBDATA/CE5ROLL/PCAML/PCAML-I-008/2B/202012/CE5-L_GRAS_PCAML-I-008_SCI_N_20201203054903_20201203054903_0004_A.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE5ROLL/PCAML/PCAML-I-008/2B/202012/CE5-L_GRAS_PCAML-I-008_SCI_N_20201203054903_20201203054903_0004_A.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE5ROLL/PCAMR/PCAMR-I-008/2B/202012/CE5-L_GRAS_PCAMR-I-008_SCI_N_20201203054903_20201203054903_0004_A.2B\n",
      "    http://moon.bao.ac.cn/cedownload/PUBDATA/CE5ROLL/PCAMR/PCAMR-I-008/2B/202012/CE5-L_GRAS_PCAMR-I-008_SCI_N_20201203054903_20201203054903_0004_A.2BL\n",
      "\n",
      "]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 指定任务为嫦娥5，载荷为LMS，数据级别留空查所有\n",
    "links = get_ChangE_download_link(1, 5, task_id='CE5', payload='', data_grade='')\n",
    "if links != None:\n",
    "    print(atool.pretty(links))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 下载数据（使用wget）\n",
    "\n",
    "- 第一步：通过`get_ChangE_download_link()`获取下载链接；\n",
    "- 第二步：将链接写入文本文件；\n",
    "- 第三步：用`wget`批量下载，当然也可以写一个函数，用`requests`下载。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    7268\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2BL\n",
      "\n",
      "]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 默认查询嫦娥4的数据，默认载荷是PCAM，默认数据级别2B\n",
    "links = get_ChangE_download_link(1, 10)\n",
    "if links != None:\n",
    "    print(atool.pretty(links))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "写入下载链接..........完成\n"
     ]
    }
   ],
   "source": [
    "# 将下载链接写入文件\n",
    "print('写入下载链接', end='')\n",
    "file_handle=open('download.txt',mode='w')\n",
    "for l in links:\n",
    "    if isinstance(l, str):\n",
    "        file_handle.write(l+'\\n')\n",
    "        print('.', end='')\n",
    "file_handle.close()\n",
    "print('完成')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: 无法创建目录 “data”: 文件已存在\n",
      "2021-05-20 01:29:18 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2B [8128512/8128512] -> \"test/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2B\" [1]\n",
      "2021-05-20 01:29:18 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2BL [6788/6788] -> \"test/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2BL\" [1]\n",
      "2021-05-20 01:29:21 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2B [8128512/8128512] -> \"test/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2B\" [1]\n",
      "2021-05-20 01:29:21 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2BL [6789/6789] -> \"test/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2BL\" [1]\n",
      "2021-05-20 01:29:24 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2B [8128512/8128512] -> \"test/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2B\" [1]\n",
      "2021-05-20 01:29:24 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2BL [6788/6788] -> \"test/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2BL\" [1]\n",
      "2021-05-20 01:29:28 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2B [8128512/8128512] -> \"test/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2B\" [1]\n",
      "2021-05-20 01:29:28 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2BL [6790/6790] -> \"test/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2BL\" [1]\n",
      "2021-05-20 01:29:31 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2B [8128512/8128512] -> \"test/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2B\" [1]\n",
      "2021-05-20 01:29:31 URL:https://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2BL [6786/6786] -> \"test/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2BL\" [1]\n",
      "下载完毕 --2021-05-20 01:29:31--\n",
      "总用时：15s\n",
      "下载了：10 个文件，15s (2.62 MB/s) 中的 39M\n"
     ]
    }
   ],
   "source": [
    "# 将数据下载到data目录中\n",
    "!mkdir data\n",
    "!wget -P test -nv -i download.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 下载数据（使用requests）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用requests下载\n",
    "import requests\n",
    "\n",
    "import os\n",
    "\n",
    "def is_datafile_exist(filename, filebuf=None, path='.', method='name'):\n",
    "    \"\"\"\n",
    "    检查给定的文件是否已经存在\n",
    "    目前只比对文件名\n",
    "    \"\"\"\n",
    "    if method == 'name':\n",
    "        for f in os.listdir(path):\n",
    "            if f[0] == '.':\n",
    "                pass\n",
    "            else:\n",
    "                if filename == f:\n",
    "                    return True\n",
    "        return False\n",
    "    elif method == 'size':\n",
    "        pass\n",
    "    elif method == 'content':\n",
    "        pass\n",
    "\n",
    "def download(url, path):\n",
    "    \"\"\"\n",
    "    将给定的url下载到path中\n",
    "    \"\"\"\n",
    "    def pleasant_size(size):\n",
    "        if size > 1024:\n",
    "            size = size/1024\n",
    "            [rs, unit] = pleasant_size(size)\n",
    "            if unit == 'B':\n",
    "                return [rs, 'kB']\n",
    "            elif unit == 'kB':\n",
    "                return [rs, 'MB']\n",
    "            elif unit == 'MB':\n",
    "                return [rs, 'GB']\n",
    "            elif unit == 'TB':\n",
    "                return [rs, 'TB']\n",
    "            elif unit == 'PB':\n",
    "                return [rs, 'PB']\n",
    "        else:\n",
    "            return [size, 'B']\n",
    "        \n",
    "        \n",
    "    filename = url[(url.rfind('/')+1):len(url)]\n",
    "    \n",
    "    r = requests.get(url, stream=True)\n",
    "    \n",
    "    if r.ok:\n",
    "        realsize = int(r.headers['Content-Length'])\n",
    "        [size, unit] = pleasant_size(realsize)\n",
    "        print('开始下载 %s（%.2f %s） ' % (filename, size, unit), end='')\n",
    "        finished = 0\n",
    "        with open(path + '/' + filename, \"wb\") as data:\n",
    "            for chunk in r.iter_content(chunk_size=1024):\n",
    "                if chunk:\n",
    "                    data.write(chunk)\n",
    "                    finished = finished + len(chunk)\n",
    "                if (finished>realsize/10):\n",
    "                    print('.', end='')\n",
    "                    finished=0\n",
    "        data.close()\n",
    "        print('完成')\n",
    "    else:\n",
    "        print(r, filename, '连接失败')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先获取下载链接。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n",
      "[\n",
      "    7268\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAMR/C/2B/2020-06/CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2BL\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2B\n",
      "    http://moon.bao.ac.cn/cedownload/CE4ROLL/CE4/PCAML/C/2B/2020-06/CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2BL\n",
      "\n",
      "]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 默认查询嫦娥4的数据，默认载荷是PCAM，默认数据级别2B\n",
    "links = get_ChangE_download_link(1, 10)\n",
    "print(len(links))\n",
    "if links != None:\n",
    "    print(atool.pretty(links))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果文件不存在（目前只判断文件名）则进行下载"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "文件 CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2B 已存在\n",
      "文件 CE4_GRAS_PCAML-C-011_SCI_N_20200428095013_20200428095013_0128_B.2BL 已存在\n",
      "文件 CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2B 已存在\n",
      "文件 CE4_GRAS_PCAMR-C-011_SCI_N_20200428094957_20200428094957_0128_B.2BL 已存在\n",
      "文件 CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2B 已存在\n",
      "文件 CE4_GRAS_PCAML-C-010_SCI_N_20200428094844_20200428094844_0128_B.2BL 已存在\n",
      "文件 CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2B 已存在\n",
      "文件 CE4_GRAS_PCAMR-C-010_SCI_N_20200428094828_20200428094828_0128_B.2BL 已存在\n",
      "文件 CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2B 已存在\n",
      "文件 CE4_GRAS_PCAML-C-009_SCI_N_20200428094715_20200428094715_0128_B.2BL 已存在\n",
      "下载完成\n",
      "CPU times: user 78.2 ms, sys: 161 ms, total: 239 ms\n",
      "Wall time: 467 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for l in links[1:len(links)]:\n",
    "    filename = l[(l.rfind('/')+1):len(l)]\n",
    "    if is_datafile_exist(filename, path='data'):\n",
    "        print('文件 %s 已存在' % (filename))\n",
    "    else:\n",
    "        download(l, 'data')\n",
    "print('下载完成')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 一次性下载全部数据文件\n",
    "\n",
    "网页端锁死了页面尺寸，一次只能下载10个数据文件。如果查询返回的数据文件数目为几千个，一页一页的下载就不太方便。\n",
    "\n",
    "使用requests可以将页面尺寸作为参数传递给GET，即可以根据需要调整页面尺寸，但是我也仅测试过每页100个条目，更大页面尺寸是否可行并没有测试过。\n",
    "\n",
    "不过既然可以通过查询知道确切的数据文件的总数目，所以只需要简单的用循环自动翻页，多次“查询-下载”即可一次性下载全部的数据文件了。\n",
    "\n",
    "比如，下面的代码将下载所有的嫦娥4号公开数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    28\n",
      "\n",
      "]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 先查一下，看看大概有多少数据\n",
    "links = get_ChangE_download_link(0, 10, task_id='CE3', payload='MUVT', data_grade='2B', date_start='2015-01-03 09:00:00', date_end='2015-01-03 10:00:00')\n",
    "if links != None:\n",
    "    print(atool.pretty(links))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "查询到数据文件共 28 个\n",
      "开始下载...\n",
      "1\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092908_20150103092908_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092803_20150103092803_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092657_20150103092657_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092552_20150103092552_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092446_20150103092446_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092341_20150103092341_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092236_20150103092236_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092150_20150103092150_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103092045_20150103092045_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091939_20150103091939_0386_B.2B（8.39 MB） .........完成\n",
      "下载完成\n",
      "2\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091834_20150103091834_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091728_20150103091728_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091623_20150103091623_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091517_20150103091517_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091412_20150103091412_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091307_20150103091307_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091221_20150103091221_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091116_20150103091116_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103091010_20150103091010_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090905_20150103090905_0386_B.2B（8.39 MB） .........完成\n",
      "下载完成\n",
      "3\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090759_20150103090759_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090654_20150103090654_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090548_20150103090548_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090443_20150103090443_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090337_20150103090337_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090252_20150103090252_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090147_20150103090147_0386_B.2B（8.39 MB） .........完成\n",
      "开始下载 CE3_BMYK_MUVT-A-0011_SCI_N_20150103090041_20150103090041_0386_B.2B（8.39 MB） .........完成\n",
      "下载完成\n",
      "CPU times: user 9.6 s, sys: 2.36 s, total: 12 s\n",
      "Wall time: 1min 32s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import time\n",
    "\n",
    "# 查第0页，不会返回具体数据，但是会返回数据文件的数目\n",
    "links = get_ChangE_download_link(0, 10, task_id='CE3', payload='MUVT', data_grade='2B', date_start='2015-01-03 09:00:00', date_end='2015-01-03 10:00:00')\n",
    "if links != None:\n",
    "    # 获取数据总条目\n",
    "    totalcount = int(links[0])\n",
    "    print('查询到数据文件共 %d 个' % (totalcount))\n",
    "    \n",
    "    print('开始下载...')\n",
    "    \n",
    "    # 每页10条数据，反复“查询-下载”\n",
    "    for i in range(int(totalcount/10+1)):\n",
    "        print(i+1)\n",
    "        links = get_ChangE_download_link(i+1, 10, task_id='CE3', payload='MUVT', data_grade='2B', date_start='2015-01-03 09:00:00', date_end='2015-01-03 10:00:00')\n",
    "        if links != None:\n",
    "                for l in links[1:len(links)]:\n",
    "                    filename = l[(l.rfind('/')+1):len(l)]\n",
    "                    if is_datafile_exist(filename, path='data'):\n",
    "                        print('文件 %s 已存在' % (filename))\n",
    "                    else:\n",
    "                        download(l, 'data')\n",
    "                print('下载完成')\n",
    "                time.sleep(0.5)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (pyqt)",
   "language": "python",
   "name": "pyqt"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}