diff --git a/notebooks/MAST/JWST/Duplication_Check/Duplication_Check.ipynb b/notebooks/MAST/JWST/Duplication_Check/Duplication_Check.ipynb new file mode 100644 index 00000000..17d096b7 --- /dev/null +++ b/notebooks/MAST/JWST/Duplication_Check/Duplication_Check.ipynb @@ -0,0 +1,499 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "# Find Existing and Planned JWST Observations Using _astroquery_" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "As with _HST, JWST_ observers are not allowed to propose observations that duplicate existing, planned, or approved observations unless they provide a scientific justification in their proposal and that request is approved. Consult the [JWST Duplicate Observation Policy](https://jwst-docs.stsci.edu/jwst-opportunities-and-policies/jwst-general-science-policies/jwst-duplicate-observations-policy) for details. Broadly speaking, observations might duplicate if they are obtained with the same scientific instrument (or a different instrument with similar configurations and capabilities), and two or more of the following apply:\n", + " * Same astrophysical source, or significant spatial overlap of fields\n", + " * Similar imaging passband, or overlapping spectral range\n", + " * Similar (spectral) resolution\n", + " * Similar exposure depth\n", + "\n", + "This notebook illustrates how to use the python package [astroquery](https://astroquery.readthedocs.io/en/latest/mast/mast.html) to search the Mikulski Archive for Space Telescopes (MAST) for potential duplicate observations. Proposers may also use the [MAST Portal](https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html) to search the archive, but that may be less efficient for large numbers of targets. \n", + "\n", + "* [Setup](#Setup)\n", + "* [Example Queries](#Example-Queries)\n", + " * [Single Target by Name](#Single-Target-by-Name)\n", + " * [Single Moving Target](#Single-Moving-Target)\n", + " * [Target Field by Position](#Target-Field-by-Position)\n", + " * [Search a Target List](#Search-for-Observations-of-Targets-in-a-List)\n", + " * [Loading Targets from a File](#Loading-Targets-from-a-File)\n", + " * [Searching with Additional Criteria](#Searching-with-Additional-Criteria)\n", + "* [Additional Resources](#Additional-Resources)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Special Disclaimer\n", + "\n", + "The capabilities described here will help identify *potential* duplications between your intended JWST observations and those that have been approved, planned, or that have already executed. \n", + "\n", + "
\n", + "\n", + "\n", + "The complete footprint of approved (but not executed), dithered, or mosaicked observations is only approximate. That is; only the primary location is reported for an observation, but not necessarily those for associated dither positions or mosaic tiles. Moreover, metadata in MAST about planned/approved observations is not sufficient to determine precisely whether your intended observation is a genuine duplication, particularly for slit or MOS spectroscopy. You are responsible for evaluating the details of the planned observations by using the accepted program's APT file (and/or the Aladin display in APT, as appropriate) to determine if the potential duplications are genuine.\n", + "\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "We begin by importing some essential python packages: general utilities in [astropy](https://www.astropy.org/), and query services in [astroquery](https://astroquery.readthedocs.io/en/latest/). We also define a utility routine to create URLs to the parent programs of matching observations." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import astropy\n", + "from astropy import table\n", + "from astropy import units as u\n", + "from astropy.table import Table\n", + "from astropy.coordinates import Angle\n", + "from astroquery.mast import Mast\n", + "from astroquery.mast import Observations\n", + "\n", + "# Give the notebook cells more of the available width, if desired:\n", + "#from IPython.display import display, HTML\n", + "#display(HTML(\"\"))\n", + "\n", + "def get_program_URL(program_id):\n", + " \"\"\"\n", + " Generate the URL for program status information, given a program ID. \n", + " \"\"\"\n", + " APT_LINK = 'https://www.stsci.edu/cgi-bin/get-proposal-info?id={}&observatory=JWST'\n", + " return APT_LINK.format(program_id)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The results of an astroquery search are contained in an [astropy table](https://docs.astropy.org/en/stable/table/). There are multiple ways to display the results; the function below displays table fields that are most relevant for identifying potential duplications of JWST observations. If this example is insufficient, you can tailor the code to your needs, using the fields listed [here](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def display_results(obs):\n", + " \"\"\"\n", + " Simple display of results related to identifying potentially duplicating targets.\n", + " Observation program title is truncated for presentation in this notebook\n", + " \"\"\"\n", + " # build the URL to the JWST programs.\n", + " obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]\n", + " obs['obs_title'] = [x[:70] for x in obs['obs_title']]\n", + " obs['obs_title'].info.format = '<'\n", + " obs['target_name', 'instrument_name', 'filters', 'dataproduct_type', 't_exptime', \n", + " 'proposal_id'].pprint(max_lines=40, max_width=90)\n", + " \n", + " print(\"\\nUnique Program Titles:\")\n", + " table.unique(obs, keys=['proposal_id'])['proposal_id','obs_title'].pprint(max_width=100)\n", + " print(\"\\nUnique URLs to status of existing programs:\")\n", + " for i in sorted(set(obs['proposal_URL'])):\n", + " print(i)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example Queries\n", + "\n", + "All of the queries below search for JWST observations, using a search radius somewhat larger than fields of view (FoV) of interest, to allow for the possibility that the FoV may be rotated when approved-but-unexecuted observations are actually scheduled. If your intended observation uses a different FoV, then adjust the search radius accordingly. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Single Target by Name\n", + "\n", + "This example shows how to query for a single target with a standard name: **Trappist-1**, which is a star with known exoplanets. The intended observations would be timeseries imaging in a small FoV. Note that the name will be resolved automatically to coordinates in this case. We use the `query_criteria()` method to limit the search to JWST observations. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "obs = Observations.query_criteria(\n", + " objectname=\"Trappist-1\", \n", + " radius=\"10s\", \n", + " obs_collection=\"JWST\"\n", + " )\n", + "print('Number of matching observations: {}'.format(len(obs)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Examine the returned table columns most relevant for identifying potential duplications. Note: it is still up to you to determine if these observations count as a duplicate with those you were planning. For instance, it does not provide the timing information necessary to determine which TRAPPIST-1 planet they are targetting. In some cases, the target name or proposal title (`obs_title`) contains this information." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "display_results(obs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Single Moving Target\n", + "\n", + "This example shows how to query for a moving target. This kind of search is limited to a modest set of solar system bodies with recognized names. Note the use of a wildcard character (**Io***) in case the target name includes other text. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "obs = Observations.query_criteria(\n", + " target_name=\"Io*\",\n", + " obs_collection=\"JWST\"\n", + " )\n", + " \n", + "display_results(obs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Target Field by Position\n", + "\n", + "This example shows how to search an area of sky for overlap with a proposed deep field. The field center (RA, Dec) is (12:12:22.513, +27:34:13.88), and the planned survey area is 30×30 arcmin. We will limit the search to JWST imaging observations. First, convert the coordinate representation to degrees, then execute the search." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "ra_deg = Angle('12:12:22.513 hours').degree\n", + "dec_deg = Angle('+27:34:13.88 degree').degree\n", + "obs = Observations.query_criteria(\n", + " s_ra=[ra_deg-0.25,ra_deg+0.25],\n", + " s_dec=[dec_deg-0.25,dec_deg+0.25],\n", + " dataproduct_type=\"image\",\n", + " obs_collection=\"JWST\"\n", + " )\n", + "\n", + "display_results(obs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There is clearly an overlap with another program, but only in certain filters." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Search for Observations of Targets in a List\n", + "\n", + "It may be best to search for individual targets (as above) with the [MAST Portal](https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html) because the results are easily visualized. But it may be more efficient to search for a large list of targets using astroquery. \n", + "\n", + "Your list might be stored in a file on your local system, and consist of coordinates and custom search radii. For simplicity, in this example the list consists of standard target names, constructed in code. Not all of the targets have approved or existing JWST observations, so the first step is to determine the number of observations for each target using the astroquery method `Observations.query_criteria_count()`.\n", + "\n", + "
\n", + "\n", + "\n", + " It is good practice to first check the number of matching observations before fetching the results themselves, in case the number of results is extremely large. This is more important when querying large MAST missions, such as HST. Note that even for a modest number of results this query may take several seconds.\n", + " \n", + "\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Create a dict to contain the number of observations for each target, initialized to zero\n", + "targets = {name:0 for name in ['CX Tau','Fomalhaut','HL Tauri','M 8','30 Dor']}\n", + "\n", + "search_radius = '30s'\n", + "for t,n in targets.items():\n", + " targets[t] = Observations.query_criteria_count(\n", + " objectname=t, \n", + " radius='{}'.format(search_radius), \n", + " obs_collection='JWST'\n", + " )\n", + "\n", + "targets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It is clear that none of the targets in the list has an excessive number of matching observations. Now check the results for the targets with non-zero matching observations in detail. Note: since the loop creates one astropy table for each search, we place each in a list and then concatenate them for display. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "obs_list = []\n", + "for t,n in targets.items():\n", + " if n > 0:\n", + " obs = Observations.query_criteria(\n", + " objectname=t, \n", + " radius='{}'.format(search_radius), \n", + " obs_collection='JWST'\n", + " )\n", + " obs_list.append(obs)\n", + " \n", + "target_matches = table.vstack(obs_list)\n", + "display_results(target_matches)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you write the results table to a disk file in ECSV format (see [astropy table I/O](https://docs.astropy.org/en/stable/io/unified.html#table-io-ascii)), that will preserve the table metadata, as well as the option for reading the file as an astropy table in a subsequent python session." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "target_matches.write('target_matches.ecsv', format='ascii.ecsv', overwrite=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Loading Targets from a File\n", + "It may be more efficient to read in a list of targets from a local file rather than manually specifying each one. You can load an example file of targets, `targets.csv` on disk using an astropy function. For convenience, this file is located in the same folder as this notebook. The file contains:\n", + "```\n", + "target_name, RA, DEC\n", + "CX Tau, 4:14:47.861, +26:48:10.91\n", + "Fomalhaut, 22:57:39.046, -29:37:20.05\n", + "HL Tau, 0:31:38.437, +18:13:57.65\n", + "M 8, 18:03:36.960, -24:23:13.20\n", + "30 Dor, 5:38:42.396, -69:06:03.36\n", + "```\n", + "The first row of the file will be interpreted as a column name in the table. This is important. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# load objects as target coordinates\n", + "targets = Table.read('targets.csv', format='ascii.csv')\n", + "targets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "With your targets in memory, you may query on them using the same ``astroquery`` functions as described above. For example, query using target names to get the count of the number of results **before** performing the query for observations in MAST.\n", + "\n", + "
\n", + "\n", + "\n", + " Query by name is not always possible: it depends on whether the name is common (can be recognized as an astrophysical source in, e.g., the Vizier catalog). You may need to query by coordinates instead.\n", + " \n", + "\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# convert names to dictionary to hold result counts\n", + "target_names = {name:0 for name in targets['target_name']}\n", + "\n", + "# get the counts of each target\n", + "search_radius = '30s'\n", + "for t,n in target_names.items():\n", + " target_names[t] = Observations.query_criteria_count(\n", + " objectname=t, \n", + " radius='{}'.format(search_radius), \n", + " dataproduct_type=\"spectrum\",\n", + " obs_collection='JWST'\n", + " )\n", + "\n", + "targets['N_obs'] = list(target_names.values())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The file contains equitorial coordinates in sexagesimal format, which need to be converted to degrees if searching on a region of sky rather than by object name. For this we use the astroquery `Angle` class." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "targets['ra_deg'] = [Angle(x+' hours').degree for x in targets['RA']]\n", + "targets['dec_deg'] = [Angle(x+' degree').degree for x in targets['DEC']]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This example will use a constant search window (i.e., spatial extent on the sky) of 0.1°, but it would be simple to customize the search area per target. This search will be for spectrum data products only, to illustrate the case where images are not important for identifying duplications. Each target search returns a separate object table, so we concatenate them for display. Note: we only search for targets where the count of observations is greater than zero to save a little time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "window = 0.05\n", + "obs_list = []\n", + "for r in targets:\n", + " if r['N_obs'] > 0:\n", + " ra_range = [r['ra_deg']-window, r['ra_deg']+window]\n", + " dec_range = [r['dec_deg']-window, r['dec_deg']+window]\n", + " obs = Observations.query_criteria(\n", + " s_ra=ra_range,\n", + " s_dec=dec_range,\n", + " dataproduct_type=\"spectrum\",\n", + " obs_collection=\"JWST\"\n", + " )\n", + " obs_list.append(obs)\n", + " \n", + "target_matches = table.vstack(obs_list)\n", + "display_results(target_matches)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Searching with Additional Criteria\n", + "\n", + "You may wish to specify in advance the filters, waveband, intstrument, or any of the other fields listed [here](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). \n", + "\n", + "As of July 2022, it is necessary to inlcude wildcard characters in the \"instrument name\" and \"filters\" fields to obtain an accurate set of results, as they sometimes include extraneous information. Additionally, there are known inconsistencies between planned and executed \"product types\" and wavelength values. These fields will be cleaned and standardized in the coming months." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "obs = Observations.query_criteria(\n", + " obs_collection=\"JWST\",\n", + " dataproduct_type=\"image\",\n", + " instrument_name=\"MIRI*\",\n", + " filters=\"F1130W*\",\n", + " t_exptime=[1000,10000]\n", + " )\n", + "display_results(obs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Additional Resources\n", + "\n", + "* [astropy](https://docs.astropy.org/en/stable/index.html) documentation\n", + "* [astroquery](https://astroquery.readthedocs.io/en/latest/mast/mast.html) documentation for querying MAST\n", + "* [Queryable fields](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html) in the MAST/CAOM database\n", + "* The [MAST Portal](https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html) web interface" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## About this notebook\n", + "\n", + "This notebook was developed by Archive Sciences Branch staff, chiefly Susan Mullally and Dick Shaw. Minor editing was provided by Thomas Dutkiewicz. For support, please contact the Archive HelpDesk at archive@stsci.edu, or through the [JWST HelpDesk Portal](https://jwsthelp.stsci.edu).\n", + "\n", + "**Last updated:** July 2022" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[Top of Page](#top)\n", + "\"Space " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.12" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/MAST/JWST/Duplication_Check/requirements.txt b/notebooks/MAST/JWST/Duplication_Check/requirements.txt new file mode 100644 index 00000000..920e6f22 --- /dev/null +++ b/notebooks/MAST/JWST/Duplication_Check/requirements.txt @@ -0,0 +1,2 @@ +astropy>=5.0 +astroquery>=0.4.6 diff --git a/notebooks/MAST/JWST/Duplication_Check/targets.csv b/notebooks/MAST/JWST/Duplication_Check/targets.csv new file mode 100644 index 00000000..dd6e5fc6 --- /dev/null +++ b/notebooks/MAST/JWST/Duplication_Check/targets.csv @@ -0,0 +1,6 @@ +target_name, RA, DEC +CX Tau, 4:14:47.861, +26:48:10.91 +Fomalhaut, 22:57:39.046, -29:37:20.05 +HL Tau, 0:31:38.437, +18:13:57.65 +M 8, 18:03:36.960, -24:23:13.20 +30 Dor, 5:38:42.396, -69:06:03.36 \ No newline at end of file diff --git a/notebooks/MAST/astroqueryMAST/beginner_search_astroqueryMAST/beginner_search_astroqueryMAST.ipynb b/notebooks/MAST/astroqueryMAST/beginner_search_astroqueryMAST/beginner_search_astroqueryMAST.ipynb new file mode 100644 index 00000000..bc67cb4f --- /dev/null +++ b/notebooks/MAST/astroqueryMAST/beginner_search_astroqueryMAST/beginner_search_astroqueryMAST.ipynb @@ -0,0 +1,395 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "# MAST API Tutorial - Astroquery API\n", + "\n", + "## Introduction and Goals:\n", + "This is a beginner tutorial on accessing the MAST database using the Astroquery API. By the end of this tutorial, you will:\n", + "\n", + "* Understand how to search for observations hosted on the MAST archive\n", + "* Download data products corresponding to your observations of interest\n", + "* Create a visual display of the downloaded data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Table of Contents\n", + "\n", + "* [Imports](#Imports)\n", + "\n", + "\n", + "* [Three ways to search for MAST observations](#Three-Ways-to-Search-for-MAST-Observations)\n", + " - [By Region](#1.-By-Region)\n", + " - [By Object Name](#2.-By-Object-Name)\n", + " - [By Criteria](#crit)\n", + " \n", + " \n", + "* [Getting Associated Data Products](#Getting-Associated-Data-Products)\n", + " - [Performing a Product Query](#Performing-a-Product-Query)\n", + " - [Filtering Data Products](#Filtering-the-Data-Products)\n", + " \n", + " \n", + "* [Downloading Products](#Downloading-Products)\n", + "\n", + "\n", + "* [Displaying Data](#Displaying-Data)\n", + "\n", + "\n", + "* [Further Reading](#Further-Reading)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Imports" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The only import we need for the MAST API is `astroquery.mast`. \n", + "\n", + "`astropy` and `matplotlib` are for opening and plotting the data, respectively." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from astroquery.mast import Observations\n", + "\n", + "from astropy.io import fits\n", + "\n", + "import matplotlib.pyplot as plt\n", + "from matplotlib.colors import SymLogNorm\n", + "%matplotlib inline " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Three Ways to Search for MAST Observations" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1. By Region\n", + "You can use `query_region` to find objects based on coordinates and radius. \n", + "The coordinates can be given as a string or `astropy.coordinates` object, and the radius as a string or float object. If no radius is specified, the default is 0.2 degrees." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# This will give a warning that the coordinates are being interpreted as an ICRS coordinate provided in degrees\n", + "obsByRegion = Observations.query_region(\"322.49324 12.16683\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# How many results? Let's look at the first ten.\n", + "print(\"Number of results:\",len(obsByRegion))\n", + "print(obsByRegion[:10])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2. By Object Name\n", + "\n", + "The `query.object` method allows you to search based on object name and optional radius argument. \n", + "The object name is first resolved to sky coordinates, and then the resolved region is queried as above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "obsByName = Observations.query_object(\"M51\",radius=\".005 deg\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Number of results:\",len(obsByName))\n", + "print(obsByName[:10])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3. By Other Criteria (with or without name/region) \n", + "\n", + "To search for observations based on additonal parameters, you can use `query_criteria`. In a sense, this is a more powerful version of the tools above, as you can still search by coordinates and objectname; however, you can inculde additional desired criteria. You may also run the search without specifying a name or coordinates.\n", + "\n", + "To perform the search, give your critera as keyword arguments. Valid criteria are listed [here](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). Some examples are: \"coordinates\", \"t_exptime\" (exposure time), and \"s_dec\" (source declination)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "obsByCriteria = Observations.query_criteria(obs_collection=[\"HLA\"], s_dec=[50,60], \n", + " calib_level=[3], proposal_pi=\"Mould*\", \n", + " dataproduct_type=\"IMAGE\", t_max=[49800,49820])\n", + "\n", + "#obsByCriteria = Observations.query_criteria(obs_id = 'hst_05766_04_wfpc2_total_wf')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false + }, + "outputs": [], + "source": [ + "obsByCriteria.show_in_notebook(display_length=5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Getting Associated Data Products" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Performing a Product Query\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Each observation returned from a MAST query can have one or more associated data products. When you input observations or observation ids (“obs_id”), `get_product_list` will return a table containing the associated data products. The product fields are documented [here](https://mast.stsci.edu/api/v0/_productsfields.html).\n", + "\n", + "Since we already have a list of observations, we can use that as the starting point for our query. To keep it simple, let's look at only the last observation from our search above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Let's select a small subset from our critera search above\n", + "newObsList = obsByCriteria[3:]\n", + "\n", + "# Now we get the list of products associated with that observation\n", + "dataProducts = Observations.get_product_list(newObsList)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false + }, + "outputs": [], + "source": [ + "dataProducts.show_in_notebook(display_length=5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Filtering the Data Products\n", + "\n", + "After the data products have been retrieved, you can use `filter_products` to download only data products that meet your given criteria. Available filters are listed [here](https://mast.stsci.edu/api/v0/_productsfields.html). Some examples are: “mrp_only” (Minimum Recommended Products) and “extension” (file extension).\n", + "\n", + "Note: the ‘AND’ operation is performed for a list of filters, and the ‘OR’ operation is performed within a filter set." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "scienceProducts = Observations.filter_products(dataProducts, productType=[\"SCIENCE\"],\n", + " mrp_only=False)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false + }, + "outputs": [], + "source": [ + "scienceProducts.show_in_notebook(display_length=5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Downloading Products\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In additon to pre-filtering, `download_products` also allows you to filter data as you request the download. In this instance, we will only download the drizzled files (drz.fits).\n", + "\n", + "Products will by default be downloaded into the current working directory, in a subdirectory called \"mastDownload.\"
\n", + "The full local file paths will have the form \"mastDownload/Mission/Observation ID/file.\"\n", + "\n", + "Note: `download_products` includes caching by default, so if the files have been downloaded, they will not be downloaded again unless caching is turned off." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "manifest = Observations.download_products(scienceProducts, mrp_only=False,\n", + " extension=(\"drz.fits\"), cache=True)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "manifest.show_in_notebook()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Displaying Data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "At this point the data is ready for analysis, and we are done using `mast.astroquery`.
\n", + "Below we take a look at the data files using `astropy` and `matplotlib`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file1 = fits.open(manifest['Local Path'][0])\n", + "file2 = fits.open(manifest['Local Path'][1])" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "f, (ax1, ax2) = plt.subplots(1, 2)\n", + "f.set_figheight(5)\n", + "f.set_figwidth(12)\n", + "ax1.imshow(file1[0].data, cmap=\"inferno\", norm=SymLogNorm(linthresh=0.03,vmin=0, vmax=1.5))\n", + "ax2.imshow(file2['SCI'].data, cmap=\"inferno\", norm=SymLogNorm(linthresh=0.03,vmin=-0.01, vmax=1.5))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Further Reading\n", + "Full documentation on **`astroquery.mast`** can be found [here](https://astroquery.readthedocs.io/en/latest/mast/mast.html)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## About this Notebook\n", + "For additonal questions, comments, or feedback, please email `archive@stsci.edu`. \n", + "\n", + "**Authors:** Thomas Dutkiewicz, Scott Fleming
\n", + "**Keywords:** MAST, astroquery
\n", + "**Created On:** 2022-06-28
\n", + "**Next Review:** 2023-06" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[Top of Page](#top)\n", + "\"Space " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.12" + }, + "widgets": { + "state": {}, + "version": "1.1.2" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} \ No newline at end of file diff --git a/notebooks/MAST/astroqueryMAST/beginner_search_astroqueryMAST/requirements.txt b/notebooks/MAST/astroqueryMAST/beginner_search_astroqueryMAST/requirements.txt new file mode 100644 index 00000000..afdd78b5 --- /dev/null +++ b/notebooks/MAST/astroqueryMAST/beginner_search_astroqueryMAST/requirements.txt @@ -0,0 +1,3 @@ +matplotlib >= 1.5.1 +astropy >= 5.0.4 +astroquery >= 0.4.6 \ No newline at end of file