\n",
"\n",
"\n",
- "The complete footprint of approved (but not executed) dithered or mosaicked observations is only approximate. That is, only the primary location is reported for an observation, but not necessarily those for associated dither positions or mosaic tiles. Moreover metadata in MAST about planned/approved observations is not sufficient to determine precisely whether your intended observation is a genuine duplication, particularly for slit or MOS spectroscopy. You are responsible for evaluating the details of the planned observations by using the accepted program's APT file (and/or the Aladin display in APT, as appropriate) to determine if the potential duplications are genuine.\n",
+ "The complete footprint of approved (but not executed) mosaicked or parallel observations is only approximate. That is, while the primary location is reported for an observation, the exact orientation is not known until the observation is scheduled for execution. Moreover metadata in MAST about planned/approved observations may not suffice to determine whether your intended observation is a genuine duplication, particularly for slit or MOS spectroscopy. You are responsible for evaluating the details of the planned observations by using the accepted program's APT file (and/or the Aladin display in APT, as appropriate) to determine if the potential duplications are genuine.\n",
"\n",
"
\n"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Table of contents\n",
+ "\n",
+ "- \n",
+ "
- Strategy \n", + "
- Setup \n", + "
- Utility Routines \n", + "
- Queries by Target/Position \n", + "
- Search by Target Name \n", + "
- Single Moving Target \n", + "
- Checking a List of Targets \n", + "
- Loading Targets from CSV \n", + "
- Evaluating Potential Duplications \n", + "
- Trappist-1 \n", + "
- V* XX Cha \n", + "
- M 57 \n", + "
- Appendix: Caveats \n", + "
- Additional Resources \n", + "
- \n",
+ "
- \n",
+ "
- \n",
+ "
- \n",
+ "
\n",
"\n",
- "All of the queries below search for JWST observations, using a search radius somewhat larger than fields of view (FoV) of interest, to allow for the possibility that the FoV may be rotated when approved-but-unexecuted observations are actually scheduled. If your intended observation uses a different FoV, then adjust the search radius accordingly. "
+ "\n",
+ "Searching for source names is not recommended when checking for potential duplications with fixed targets because the matches are not reliable, consistent, or complete. The exception would be for matching Object names of well known solar system bodies.\n",
+ "\n",
+ "The best approach is to use an independent source for the coordinates of your desired target. Failing that, the Mast.resolve_object() method will return coordinates in the ICRS reference frame, which you can use to verify whether a target name resolves to the coordinates you intend. See the Appendix for details.\n",
+ "\n",
+ "
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- "### Single Target by Name\n",
+ "\n",
+ "### Search by Target Name\n",
"\n",
- "This example shows how to query for a single target with a standard name, **Trappist-1** which is a star with a known exo-planet. The intended observations would be timeseries imaging in a small FoV. Note that the name will be resolved automatically to coordinates in this case. We use the query_criteria() method to limit the search to JWST observations. "
+ "This example shows how to query for a single target with a standard name, **HD 104237** which is a T-Tauri star. We first resolve the name and verify that the coordinates are correct. "
]
},
{
@@ -146,19 +226,17 @@
"metadata": {},
"outputs": [],
"source": [
- "obs = Observations.query_criteria(\n",
- " objectname=\"Trappist-1\", \n",
- " radius=\"10s\", \n",
- " obs_collection=\"JWST\"\n",
- " )\n",
- "print('Number of matching observations: {}'.format(len(obs)))"
+ "# Resolve the coordinates if necessary\n",
+ "coords = Mast.resolve_object('HD 104237')\n",
+ "coords"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "Examine the returned table columns most relevant for identifying potential duplications. Note: it is still up to you to determine if these observations count as a duplicate with those you were planning. For instance, it does not provide the timing information necessary to determine which TRAPPIST-1 planet they are targetting. In some cases, the target name or proposal title (obs_title) contains this information."
+ "In this case the resolved coordinates agree with those in (see [Simbad](https://simbad.u-strasbg.fr/simbad/sim-basic?Ident=HD+104237&submit=SIMBAD+search)), so it is safe to execute a simple **cone search** (a search within a specified radius of a place on the sky) to find JWST existing or planned observations. \n",
+ "The [astroquery.mast](https://astroquery.readthedocs.io/en/latest/mast/mast.html) package provides the method **Observations.query_criteria()** to specify the parameters for the search; provide them as key=value pairs. The full set of query parameters for this method may be found on [CAOM Field Descriptions](https://mast.stsci.edu/api/v0/_c_a_o_mfields.html). "
]
},
{
@@ -167,7 +245,25 @@
"metadata": {},
"outputs": [],
"source": [
- "display_results(obs)"
+ "# If the coordinates are ok\n",
+ "obs = Observations.query_criteria(\n",
+ " obs_collection='JWST'\n",
+ " ,objectname='HD 104237'\n",
+ " ,radius='20s'\n",
+ " )\n",
+ "print('Number of matching observations: {}'.format(len(obs)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ " The number of matches is zero, so there are no potential duplications with this target.\n",
+ "\n",
+ "
"
]
},
{
@@ -177,7 +273,7 @@
"\n",
"### Single Moving Target\n",
"\n",
- "This example shows how to query for a moving target. This kind of search is limited to a modest set of solar system bodies with recognized names. Note the use of a wildcard character (*) in case the target name includes other text."
+ "Moving targets, by definition, do not lend themselves to searches by position. This kind of search is limited to a modest set of solar system bodies with recognized names. Note the use of a wildcard character (*) in case the target name includes other text."
]
},
{
@@ -191,17 +287,14 @@
" obs_collection=\"JWST\"\n",
" )\n",
" \n",
- "display_results(obs)"
+ "print('Number of matching observations: {}'.format(len(obs)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- "### Target Field by Position\n",
- "\n",
- "This example shows how to search an area of sky for overlap with a proposed deep field. The field center (RA, Dec) is (12:12:22.513, +27:34:13.88), and the planned survey area is 30×30 arcmin. We will limit the search to JWST imaging observations. First, convert the coordinate representationn to degrees, then execute the search."
+ "There are some JWST observations of Io, including some that are planned but have not executed as of the beginning of Cy-2 (i.e., `calib_level = -1`). The details are in the returned table of results, the most essential of which can be viewed below."
]
},
{
@@ -210,30 +303,41 @@
"metadata": {},
"outputs": [],
"source": [
- "ra_deg = Angle('12:12:22.513 hours').degree\n",
- "dec_deg = Angle('+27:34:13.88 degree').degree\n",
- "obs = Observations.query_criteria(\n",
- " s_ra=[ra_deg-0.25,ra_deg+0.25],\n",
- " s_dec=[dec_deg-0.25,dec_deg+0.25],\n",
- " dataproduct_type=\"image\",\n",
- " obs_collection=\"JWST\"\n",
- " )"
+ "out_cols = ['target_name','instrument_name','filters','calib_level','t_exptime','proposal_id']\n",
+ "obs[out_cols].show_in_notebook()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "There is clearly an overlap with another program, but only in certain filters:"
+ "You may need to examine the specifics of the programs that obtained the observations to know whether your intended observations would duplicate. Here are the unique Program Titles and URLs for the program status pages, which may offer clues:"
]
},
{
"cell_type": "code",
"execution_count": null,
- "metadata": {},
+ "metadata": {
+ "scrolled": true
+ },
"outputs": [],
"source": [
- "display_results(obs)"
+ "obs['title'] = [x[:80] for x in obs['obs_title']]\n",
+ "unique(obs['proposal_id','title']).pprint(max_width=-1)\n",
+ "obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]\n",
+ "unique(obs['proposal_id','proposal_URL']).pprint(max_width=-1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ " There are images, spectra and IFU observations of this target that could conflict with your intended proposal. Proposed images in passbands that overlap the IFU data would have to be examined closely for duplication. \n",
+ "\n",
+ "
"
]
},
{
@@ -241,16 +345,68 @@
"metadata": {},
"source": [
"\n",
- "### Search for Observations of Targets in a List\n",
+ "## Checking a List of Targets\n",
"\n",
- "It may be best to search for individual targets (as above) with the [MAST Portal](https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html) because the results are easily visualized. But it may be more efficient to search for a large list of targets using astroquery. \n",
+ "It is often useful to search for individual targets with the [MAST Portal](https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html) because the results are easily visualized. But it is more efficient to search over a large number of targets using astroquery.mast. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### Loading Targets from CSV\n",
+ "For efficiency, make a CSV (comma-separated variable) list of your targets, one per row. The list can alternatively be read from a local file: just substitute the file name for `targ_table` in the first argument of the `Table.read()` method. The list can contain many fields, but at a minimum must contain the target name and the equitorial coordinates. \n",
"\n",
- "Your list might be stored in a file on your local system, and consist of coordinates and custom search radii. But for simplicity the list in this example consists of standard target names, constructed in code. Not all of the targets have approved or existing JWST observations, so the first step is to determine the number of observations for each target using the astroquery method Observations.query_criteria_count().\n",
+ "The first row of the file will be interpreted as a column name in the table. This is important. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "targ_table = '''\n",
+ "target_name, RA, DEC, Range, Description\n",
+ "Trappist-1, 23:06:29.3684948589, -05:02:29.037301866, 20s, Exoplanet host star\n",
+ "V* XX Cha, 11:11:39.559, -76:20:15.04, 20s, Variable star with debris disk\n",
+ "M 31, 00:42:44.330, +41:16:07.50, 24.0m, Andromeda Galaxy\n",
+ "M 57, 18:53:35.0967659112, +33:01:44.883287544, 4m, Planetary nebula\n",
+ "'''\n",
+ "targets = Table.read(targ_table, format='ascii.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Next, make new columns to hold the coordinates and ranges in units of degrees. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "targets['ra_deg'] = [Angle(x+' hours').degree for x in targets['RA']]\n",
+ "targets['dec_deg'] = [Angle(x+' degree').degree for x in targets['DEC']]\n",
+ "targets['range_deg'] = [Angle(x).degree for x in targets['Range']]\n",
+ "targets['N_obs'] = 0 # field to hold the count of matched observations \n",
+ "targets"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It can save some effort to determine the number of JWST Observations that match each of your targets using the query_criteria_count() method. Note that the coordinate parameters must be specified as lower and upper bounds of a range (i.e., a python list), so use the `coord_ranges()` utility function.\n",
"\n",
"\n",
"\n",
"\n",
- " It is good practice to first check the number of matching observations before fetching the results themselves, in case the number of results is extremely large. This is more important when querying large MAST missions, such as HST. Note that even for a modest number of results this query may take several seconds.\n",
+ "It is good practice to first check the number of matching observations before fetching the results themselves, in case the number of results is very large. This is more important when querying MAST missions with a very large number of observations, such as HST. This type of search is typically fairly fast.\n",
" \n",
"\n",
"
"
@@ -262,29 +418,39 @@
"metadata": {},
"outputs": [],
"source": [
- "# Create a dict to contain the number of observations for each target, initialized to zero\n",
- "targets = {name:0 for name in ['CX Tau','Fomalhaut','HL Tauri','M 8','30 Dor']}\n",
- "\n",
- "# Set out parameters for this search\n",
- "search_radius = '30s'\n",
- "mission = \"JWST\"\n",
+ "# get the counts of each target\n",
+ "for t in targets:\n",
+ " (ra_range, dec_range) = coord_ranges(t['ra_deg'],t['dec_deg'], t['range_deg'], t['range_deg'])\n",
+ " t['N_obs'] = Observations.query_criteria_count(\n",
+ " obs_collection='JWST'\n",
+ " ,s_ra=ra_range\n",
+ " ,s_dec=dec_range\n",
+ " )\n",
"\n",
- "# Go through the targets, setting the number of results for each dictionary item\n",
- "for t,n in targets.items():\n",
- " targets[t] = Observations.query_criteria_count(\n",
- " objectname = t\n",
- " , radius = search_radius\n",
- " , obs_collection = mission\n",
- " )\n",
+ "targets['target_name','N_obs'].show_in_notebook()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
"\n",
- "targets"
+ "\n",
+ "For targets where no JWST Observations exist, there is no potential for duplication. \n",
+ "\n",
+ "
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "It is clear that none of the targets in the list has an excessive number of matching observations. Now check the results for the targets with non-zero matching observations in detail. Note: since the loop creates one astropy table for each search, we place each in a list and then concatenate them for display. "
+ "\n",
+ "## Evaluating Potential Duplications\n",
+ "\n",
+ "To examine the matching Observations for the other targets in more detail, use the query_criteria()\n",
+ "method. "
]
},
{
@@ -293,28 +459,50 @@
"metadata": {},
"outputs": [],
"source": [
- "# Create a list to contain the observations themselves\n",
- "obs_list = []\n",
+ "targets['obs'] = None\n",
+ "for t in targets:\n",
+ " if t['N_obs'] > 0:\n",
+ " (ra_range, dec_range) = coord_ranges(t['ra_deg'],t['dec_deg'], t['range_deg'], t['range_deg'])\n",
+ " t['obs'] = Observations.query_criteria(\n",
+ " obs_collection=\"JWST\"\n",
+ " ,s_ra=ra_range\n",
+ " ,s_dec=dec_range\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Targets with matching JWST Observations are examined in the following sub-sections."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### Trappist-1\n",
"\n",
- "# Same search as above, but get the results instead of the count\n",
- "for t,n in targets.items():\n",
- " if n > 0:\n",
- " obs = Observations.query_criteria(\n",
- " objectname = t\n",
- " , radius = search_radius \n",
- " , obs_collection = mission\n",
- " )\n",
- " obs_list.append(obs)\n",
- " \n",
- "target_matches = table.vstack(obs_list)\n",
- "display_results(target_matches)"
+ "Trappist-1 is a well known exo-planet host star. If the intended observations are timeseries spectroscopy, the search would naturally be limited to a small area of sky. There are quite a few JWST observations of this target, so it is important to examine the relevant details. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "out_cols = ['target_name','instrument_name','filters','t_exptime','proposal_id']\n",
+ "obs = targets[0]['obs']\n",
+ "unique(obs[out_cols]).show_in_notebook()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "If you write the results table to a disk file in ECSV format (see [astropy table I/O](https://docs.astropy.org/en/stable/io/unified.html#table-io-ascii)), that will preserve the table metadata, as well as the option for reading the file as an astropy table in a subsequent python session."
+ "You will need to examine the specifics of the programs that obtained the observations to know for certain which of the 7 known exoplanets was being observed, and therefore whether your intended observations would duplicate. Here are the unique Program Titles, which may offer clues:"
]
},
{
@@ -323,25 +511,15 @@
"metadata": {},
"outputs": [],
"source": [
- "target_matches.write('target_matches.ecsv', format='ascii.ecsv', overwrite=True)"
+ "obs['title'] = [x[:80] for x in obs['obs_title']]\n",
+ "unique(obs['proposal_id','title']).pprint(max_width=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "\n",
- "### Loading Targets from a File\n",
- "It may be more efficient to read in a list of targets from a local file rather than manually specifying each one. You can load an example file of targets, `targets.csv` on disk using an astropy function. The file contains:\n",
- "```\n",
- "target_name, RA, DEC\n",
- "CX Tau, 4:14:47.861, +26:48:10.91\n",
- "Fomalhaut, 22:57:39.046, -29:37:20.05\n",
- "HL Tau, 0:31:38.437, +18:13:57.65\n",
- "M 8, 18:03:36.960, -24:23:13.20\n",
- "30 Dor, 5:38:42.396, -69:06:03.36\n",
- "```\n",
- "The first row of the file will be interpreted as a column name in the table. This is important. "
+ "The Program Status Pages offer the complete specifications for each existing program:"
]
},
{
@@ -350,121 +528,135 @@
"metadata": {},
"outputs": [],
"source": [
- "# load objects as target coordinates\n",
- "targets = Table.read('targets.csv', format='ascii.csv')\n",
- "targets"
+ "obs['proposal_URL'] = [get_program_URL(x) for x in obs['proposal_id']]\n",
+ "unique(obs['proposal_id','proposal_URL']).pprint(max_width=-1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "With your targets in memory, you may query on them using the same ``astroquery`` functions as described above. For example, query by source identifier using of target names to get the count of the number of results before performing the query for observations in MAST.\n",
- "\n",
- "\n",
+ "![\"Space](\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\")
\n",
"\n",
- "Last updated: August 2023\n",
- "\n",
- "\n",
- ""
+ "**Last update:** 2023 Aug"
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {
@@ -513,7 +708,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.4"
+ "version": "3.10.12"
}
},
"nbformat": 4,
diff --git a/notebooks/JWST/duplication_checking/duplication_pure_python.ipynb b/notebooks/JWST/duplication_checking/duplication_pure_python.ipynb
deleted file mode 100644
index 50a32b55..00000000
--- a/notebooks/JWST/duplication_checking/duplication_pure_python.ipynb
+++ /dev/null
@@ -1,1059 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- " search radius is needed for position queries\"\n",
- " raise TypeError(err)\n",
- " elif coordinates is None:\n",
- " service = \"Mast.Caom.Filtered\"\n",
- " else:\n",
- " print(\"'submit_mashup_request()' expects coordinates in a tuple\")\n",
- " return\n",
- " \n",
- " # Update settings for retrieving full query results or results count.\n",
- " if count:\n",
- " columns = \"COUNT_BIG(*)\" # \"COUNT_BIG(*)\" will only get a count of the results\n",
- " else:\n",
- " columns = \"*\"\n",
- " \n",
- " # Construct the mashup request.\n",
- " params_dict = {\"columns\": columns, \"filters\": filt_list}\n",
- " if coordinates:\n",
- " position = \"{0}, {1}, {2}\".format(ra_deg, dec_deg, radius)\n",
- " params_dict[\"position\"] = position\n",
- " mashupRequest = {\"service\":service,\n",
- " \"format\":\"json\",\n",
- " \"params\":params_dict\n",
- " }\n",
- "\n",
- " # Execute the query.\n",
- " headers, outString = mastQuery(mashupRequest)\n",
- " queryResults = json.loads(outString)\n",
- " \n",
- " # Analyze the results count if counting.\n",
- " if count:\n",
- " data = queryResults['data']\n",
- " result_count = data[0]['Column1']\n",
- " if result_count == 0:\n",
- " raise RuntimeError(\"No matching results found\")\n",
- " elif result_count > 50000:\n",
- " raise RuntimeError(\"Query returned {} results (>50k limit)\".format(result_count))\n",
- " else:\n",
- " # Rerun the full query if result counts are acceptable.\n",
- " return submit_mashup_request(filters, \n",
- " coordinates=coordinates,\n",
- " radius=radius,\n",
- " count=False\n",
- " )\n",
- " # Otherwise return the query results.\n",
- " else:\n",
- " print(\"Returned {} query results\".format(len(queryResults['data'])))\n",
- " return queryResults"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- ""
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/notebooks/JWST/duplication_checking/targets.csv b/notebooks/JWST/duplication_checking/targets.csv
deleted file mode 100644
index dd6e5fc6..00000000
--- a/notebooks/JWST/duplication_checking/targets.csv
+++ /dev/null
@@ -1,6 +0,0 @@
-target_name, RA, DEC
-CX Tau, 4:14:47.861, +26:48:10.91
-Fomalhaut, 22:57:39.046, -29:37:20.05
-HL Tau, 0:31:38.437, +18:13:57.65
-M 8, 18:03:36.960, -24:23:13.20
-30 Dor, 5:38:42.396, -69:06:03.36
\ No newline at end of file
\n",
"\n",
"\n",
- " Query by name is not always possible: it depends on whether the name is common (can be recognized as an astrophysical source in, e.g., the Vizier catalog). You may need to query by coordinates instead.\n",
- " \n",
+ " There are both NIRISS and NIRSpec spectroscopic observations of this target that could conflict with your intended proposal. \n",
"\n",
"
"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### V* XX Cha\n",
+ "\n",
+ "This pre-main-sequence star has a debris disk, and might be worth a coronagraphic observation. The search area is small (20 arcsec) to exclude nearby targets."
+ ]
+ },
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
- "# convert names to dictionary to hold result counts\n",
- "target_names = {name:0 for name in targets['target_name']}\n",
- "\n",
- "# Search parameters\n",
- "search_radius = '30s'\n",
- "mission = 'JWST'\n",
- "dpt = 'spectrum'\n",
+ "out_cols = ['target_name','instrument_name','filters','t_exptime','calib_level','proposal_id']\n",
+ "obs = targets[1]['obs']\n",
+ "unique(obs[out_cols]).show_in_notebook()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that some of the observations for this target had not been observed (at the beginning of Cy-2). "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
"\n",
- "# Try to get the target names, but gracefully handle failures\n",
- "for t,n in target_names.items():\n",
- " target_names[t] = Observations.query_criteria_count(\n",
- " objectname = t\n",
- " , radius = search_radius\n",
- " , obs_collection = mission\n",
- " , dataproduct_type = dpt\n",
- " )\n",
- " \n",
- " \n",
- "targets['N_obs'] = list(target_names.values())"
+ "\n",
+ "While JWST spectra from MIRI and NIRSpec exist or are planned, there is no potential for duplication with a coronagraphic observation. \n",
+ "\n",
+ "
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "The file contains equatorial coordinates in sexagesimal format, which need to be converted to degrees if searching on a region of sky rather than by object name. For this we use the astroquery Angle class."
+ "\n",
+ "### M 57\n",
+ "\n",
+ "The Ring Nebula (NGC 6720) is well observed, particularly with HST and, recently, with JWST. We searched a 4-arcmin region around the source to include potential spectral Observations in the nebular periphery. The following table shows the unique combinations of instrument configurations. "
]
},
{
"cell_type": "code",
"execution_count": null,
- "metadata": {
- "tags": []
- },
+ "metadata": {},
"outputs": [],
"source": [
- "targets['ra_deg'] = [Angle(x+' hours').degree for x in targets['RA']]\n",
- "targets['dec_deg'] = [Angle(x+' degree').degree for x in targets['DEC']]"
+ "out_cols = ['target_name','instrument_name','filters','t_exptime','proposal_id']\n",
+ "obs = targets[3]['obs']\n",
+ "unique(obs[out_cols]).show_in_notebook()"
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "tags": []
- },
- "outputs": [],
+ "cell_type": "markdown",
+ "metadata": {},
"source": [
- "targets"
+ "\n",
+ "\n",
+ "\n",
+ " There are both MIRI and NIRCam images in many passbands, and MIRI and NIRSpec spectroscopic IFU Observations of this target that could conflict with the intended proposal. \n",
+ "\n",
+ "
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "This example will use a constant search window (i.e., spatial extent on the sky) of 0.1°, but it would be simple to customize the search area per target. This search will be for spectrum data products only, to illustrate the case where images are not important for identifying duplications. Each target search returns a separate object table, so we concatenate them for display. Note: we only search for targets where the count of observations is greater than zero to save a little time."
+ "To retain this target it would be necessary to obtain spectra in a different location, or to use a different instrument/configuration/data-taking mode (e.g., time-series of the central star). It may be helpful to see the [footprints of these Observations](https://mast.stsci.edu/portal/Mashup/Clients/Mast/Portal.html?searchQuery=%7B%22service%22%3A%22CAOMFILTERED%22%2C%22inputText%22%3A%5B%7B%22paramName%22%3A%22obs_collection%22%2C%22niceName%22%3A%22obs_collection%22%2C%22values%22%3A%5B%22JWST%22%5D%2C%22valString%22%3A%22JWST%22%2C%22isDate%22%3Afalse%2C%22separator%22%3A%22%3B%22%2C%22facetType%22%3A%22discrete%22%2C%22displayString%22%3A%22JWST%22%7D%5D%2C%22position%22%3A%22283.39623652463%2C%2033.02913424654%2C%200.066667%22%2C%22paramsService%22%3A%22Mast.Caom.Filtered.Position%22%2C%22title%22%3A%22MAST%3A%20%20Advanced%20Search%201%22%2C%22tooltip%22%3A%22JWST%3B%20%20m%2057%20r%3D4m%3B%22%2C%22ra%22%3A283.39623652463%2C%22dec%22%3A33.02913424654%2C%22radius%22%3A0.066667%2C%22columns%22%3A%22*%22%2C%22columnsConfig%22%3A%22Mast.Caom.Cone%22%7D) in the MAST Portal, or to view the observation specifications in Program [GO-1558](https://www.stsci.edu/cgi-bin/get-proposal-info?id=1558&observatory=JWST)."
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "# Search parameters\n",
- "window = 0.05\n",
- "mission = 'JWST'\n",
- "dpt = 'spectrum'\n",
- "\n",
- "# Initialize list to hold obs\n",
- "obs_list = []\n",
- "\n",
- "# \n",
- "for r in targets:\n",
- " if r['N_obs'] > 0:\n",
- " RA = r['ra_deg']\n",
- " DEC = r['dec_deg']\n",
- " ra_range = [RA-window, RA+window]\n",
- " dec_range = [DEC-window, DEC+window]\n",
- " obs = Observations.query_criteria(\n",
- " s_ra = ra_range\n",
- " , s_dec = dec_range\n",
- " , dataproduct_type = dpt\n",
- " , obs_collection = mission\n",
- " )\n",
- " obs_list.append(obs)\n",
- " \n",
- "target_matches = table.vstack(obs_list)\n",
- "display_results(target_matches)"
+ "\n",
+ "## Appendix: Caveats"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It is possible to search MAST for individual sources by name, in one of two ways: *Object Name* or by *Target Name*. \n",
+ "\n",
+ "- **Object Name:** MAST invokes an astrophysical name resolver (e.g., [NED](https://ned.ipac.caltech.edu/byname) or [Simbad](https://simbad.u-strasbg.fr/simbad/sim-fid)) to look up source coordinates. Not all object names are recognized by resolvers. Also, different resolvers sometimes return substantially different coordinates; MAST will pick one of them (but not tell you which one). Sometimes the coordinates returned by the resolvers differ by substantial amounts, often because they refer to different sources.\n",
+ " - A classic case is the bright star μ Eri from the Bayer Greek letter system, and the star MU Eri in the general catalog of variable stars. in MAST,\n",
+ " - Searching for `* mu. eri` (or just plain `mu eri` matches **μ Eri** (RA = 4:45:30.15, Dec = -3:15:16.777)\n",
+ " - Searching for `V* mu eri` matches **MU Eri** (RA = 2:48:10.566, Dec = -15:18:04.03)\n",
+ "- **Target Name:** This search matches to target names specified by Investigators in observing proposals to refer to sources. These are not guaranteed to match standard names for astrophysical sources, and often do not."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "Searching by source names is not recommended when checking for potential duplications with fixed targets because the matches are not reliable, consistent, or complete. The exception would be for matching Object names of well known solar system bodies.\n",
+ "\n",
+ "The best approach is to use an independent source for the coordinates of your desired target. Failing that, the Mast.resolve_object() method will return coordinates in the ICRS reference frame, which you can use to verify whether a target name resolves to the coordinates you intend. See the Appendix for details.\n",
+ "\n",
+ "
"
]
},
{
@@ -472,7 +664,7 @@
"metadata": {},
"source": [
"\n",
- "# Additional Resources\n",
+ "## Additional Resources\n",
"\n",
"* [astropy](https://docs.astropy.org/en/stable/index.html) documentation\n",
"* [astroquery](https://astroquery.readthedocs.io/en/latest/mast/mast.html) documentation for querying MAST\n",
@@ -486,15 +678,18 @@
"source": [
"## About this notebook\n",
"\n",
- "This notebook was developed by Archive Sciences Branch staff, chiefly Susan Mullally and Dick Shaw. Additional editing was provided by Thomas Dutkiewicz. \n",
- "\n",
- "For support, please contact the Archive HelpDesk at archive@stsci.edu.\n",
+ "This notebook was developed by Archive Sciences Branch staff. For support, please contact the Archive HelpDesk at archive@stsci.edu, or through the [JWST HelpDesk Portal](https://jwsthelp.stsci.edu). \n",
+ "\n",
"\n",
- "Last updated: August 2023\n",
- "\n",
- "\n",
- ""
+ "**Last update:** 2023 Aug"
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {
@@ -513,7 +708,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.4"
+ "version": "3.10.12"
}
},
"nbformat": 4,
diff --git a/notebooks/JWST/duplication_checking/duplication_pure_python.ipynb b/notebooks/JWST/duplication_checking/duplication_pure_python.ipynb
deleted file mode 100644
index 50a32b55..00000000
--- a/notebooks/JWST/duplication_checking/duplication_pure_python.ipynb
+++ /dev/null
@@ -1,1059 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Using the API to Query Planned JWST Observations in MAST
\n", - "\n", - "In this Notebook we'll take a look at how to use the MAST API to find planned JWST observations from GTO and ERS programs. We'll start with a basic filtered query to find all planned observations before moving on to constructing more specific requests:
\n", - "- \n",
- "
- Searching by position \n", - "
- Analyzing our results \n", - "
- Saving results to a file \n", - "
- Querying multiple positions at once \n", - "
- Resolving stationary target names \n", - "
- Searching for moving target names \n", - "\n", - "
Be aware that MAST queries can take some time to run and get a response, so please be patient when running cells back to back. If you encounter any errors, try to run the cell again when the previous cells have completed execution.
\n", - "IMPORTANT DISCLAIMER: To avoid unintentional duplications, JWST proposers are required to check their proposed observations against those already approved. The tools provided here may be able to assist a user in identifying POTENTIAL observing conflicts for JWST programs. It remains the PI's responsibility to determine whether or not the result provided by these tools constitutes an actual conflict, in accordance with the JWST Duplicate Observations Policy.
\n", - "(Much of this code is based on the tutorials and examples provided in the MAST API documentation pages found here)
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Standard import statements for a MAST Mashup API request
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "import os\n", - "import time\n", - "import re\n", - "import json\n", - "\n", - "try: # Python 3.x\n", - " from urllib.parse import quote as urlencode\n", - " from urllib.request import urlretrieve\n", - "except ImportError: # Python 2.x\n", - " from urllib import pathname2url as urlencode\n", - " from urllib import urlretrieve\n", - "\n", - "try: # Python 3.x\n", - " import http.client as httplib \n", - "except ImportError: # Python 2.x\n", - " import httplib \n", - "\n", - "from astropy.table import Table\n", - "import numpy as np\n", - "\n", - "import pprint\n", - "pp = pprint.PrettyPrinter(indent=4)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Define the MAST query module to handle appropriate formatting
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def mastQuery(request):\n", - " \"\"\"Perform a MAST query.\n", - " \n", - " Parameters\n", - " ----------\n", - " request : dict\n", - " The MAST request json object\n", - " \n", - " Returns\n", - " -------\n", - " head, content \n", - " Head is the response HTTP headers, and content is the returned data\n", - " \"\"\"\n", - " \n", - " server='mast.stsci.edu'\n", - "\n", - " # Grab Python Version \n", - " version = \".\".join(map(str, sys.version_info[:3]))\n", - "\n", - " # Create Http Header Variables\n", - " headers = {\"Content-type\": \"application/x-www-form-urlencoded\",\n", - " \"Accept\": \"text/plain\",\n", - " \"User-agent\":\"python-requests/\"+version}\n", - "\n", - " # Encoding the request as a json string\n", - " requestString = json.dumps(request)\n", - " requestString = urlencode(requestString)\n", - " \n", - " # opening the https connection\n", - " conn = httplib.HTTPSConnection(server)\n", - "\n", - " # Making the query\n", - " conn.request(\"POST\", \"/api/v0/invoke\", \"request=\"+requestString, headers)\n", - "\n", - " # Getting the response\n", - " resp = conn.getresponse()\n", - " head = resp.getheaders()\n", - " content = resp.read().decode('utf-8')\n", - "\n", - " # Close the https connection\n", - " conn.close()\n", - "\n", - " return head,content" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here is where we begin to customize the code to perform our own queries:
\n", - "The first search below sends a basic filtered query to count all planned observations, designated by a calibration level of -1. In the 'data' entry, we can see 'Column1' returns with the number of results for the submitted query.
\n", - "- \n",
- "
- \"paramName\":\"calib_level\" allows us to filter based on calibration level. \n", - "
- \"values\":[\"-1\"] defines a planned observation. \n", - "
Additional parameters to filter on:
\n", - "- \n",
- "
- target_name \n", - "
- s_ra \n", - "
- s_dec \n", - "
- instrument \n", - "
- filters \n", - "
- proposal_id \n", - "
- target_classification \n", - "
A succesful query should return a dictionary with a 'status': 'COMPLETE' entry. The 'data' from this query should simply contain a count of all the planned observations for JWST present in MAST's CAOM (Common Archive Observation Model) database.
\n", - "\n", - "Next, we will establish a script that will construct a Mashup request given a number of different filters.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def submit_mashup_request(filters, coordinates=None, radius=0.2, count=True):\n", - " \"\"\"\n", - " Construct a filtered mashup request to send to the mastQuery function. Return \n", - " either the results of the query or just the results count.\n", - " \n", - " Parameters\n", - " ----------\n", - " filters : dict\n", - " A dictionary of query filters to use in a mashup request.\n", - " coordinates : tuple, optional\n", - " Expects a pair of coordinates in degrees, if querying a position.\n", - " (default=None)\n", - " radius : float, optional\n", - " Defines the radius to search around the designated coordinates within.\n", - " Also in degrees. (default=0.2)\n", - " count : bool, optional\n", - " Flag to designate whether a full query is submitted, or just the count\n", - " results. (default=True)\n", - " \n", - " Returns\n", - " -------\n", - " queryResults : dict\n", - " A dictionary of query results with keys of 'data', 'fields', 'msg',\n", - " 'paging', and 'status'.\n", - " \"\"\"\n", - " \n", - " # Construct the list of filters to query on.\n", - " filt_list = []\n", - " for f, val in filters.items():\n", - " vals = val if isinstance(val, list) else [val]\n", - " entry = {\"paramName\": f,\n", - " \"values\": vals\n", - " }\n", - " if f == \"filters\":\n", - " entry[\"seperators\"] = \";\"\n", - " elif f == \"target_name\":\n", - " entry[\"values\"] = []\n", - " entry[\"freeText\"] = \"%{}%\".format(val)\n", - " filt_list.append(entry)\n", - " \n", - " # Update settings if coordinates are provided.\n", - " if isinstance(coordinates, tuple):\n", - " service = \"Mast.Caom.Filtered.Position\"\n", - " ra_deg, dec_deg = coordinates\n", - " if not isinstance(radius, float):\n", - " err = \"AThis function allows us to easily retrieve all planned JWST observation information (as long as there are fewer than 50k) with the following:
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "MY_FILTS = {\"calib_level\": \"-1\", \"obs_collection\": \"JWST\"}\n", - "ALL_RESULTS = submit_mashup_request(MY_FILTS)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "Searching by position
\n", - "Next, we can begin adding additional filters to narrow our query, starting with position. First off, in order to send a filtered position search via the API, we'll need to format the position in question as an RA, DEC tuple in degrees. Converting a set of coordinates to degrees is made pretty easy by using the astropy.coordinates SkyCoord class.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from astropy.coordinates import SkyCoord\n", - "\n", - "def check_user_coordinates(*args):\n", - " \"\"\"\n", - " Convert a single user coordinate pair to a tuple of degrees. Able to accept multiple input formats, \n", - " relying on SkyCoord class.\n", - " \n", - " Parameters\n", - " ----------\n", - " *args : multiple\n", - " A user may pass individual RA and DEC coordinates or as a single tuple.\n", - " Coordinates may be floats or strings.\n", - " \n", - " Returns\n", - " -------\n", - " coords : tuple\n", - " An RA, DEC tuple in degrees.\n", - " \"\"\"\n", - " \n", - " # Extract RA and Dec values from args.\n", - " if len(args) == 1 and isinstance(args[0], tuple):\n", - " ra, dec = args[0]\n", - " elif len(args) == 2:\n", - " ra = args[0]\n", - " dec = args[1]\n", - " else:\n", - " err = \"Could not parse coordinates: {}\".format(args)\n", - " raise ValueError(err)\n", - " \n", - " # Construct the tuple of formatted coordinates.\n", - " if isinstance(ra, str) and isinstance(dec, str):\n", - " ra = format_coordinate_str(ra, ['h', 'm', 's'])\n", - " dec = format_coordinate_str(dec, ['d', 'm', 's'])\n", - " obj = SkyCoord(ra, dec)\n", - " ra_deg = obj.ra.deg\n", - " dec_deg = obj.dec.deg\n", - " coords = (ra_deg, dec_deg)\n", - " elif isinstance(ra, float) and isinstance(dec, float):\n", - " coords = (ra, dec)\n", - " else:\n", - " err = \"Could not parse RA/DEC: {0}, {1}\".format(ra, dec)\n", - " raise TypeError(err)\n", - " \n", - " return coords\n", - "\n", - "def format_coordinate_str(coord, units):\n", - " \"\"\"\n", - " Make sure coordinate strings are in \"**h**m**s\"-like format.\n", - " \n", - " Parameters\n", - " ----------\n", - " coord : str\n", - " An RA or Dec coordinate string.\n", - " units : list\n", - " A list of unit letters, either [h, m, s] or [d, m, s].\n", - " \n", - " Returns\n", - " -------\n", - " new : str\n", - " A formatted coordinate string.\n", - " \"\"\"\n", - " \n", - " assert isinstance(coord, str)\n", - " new = \"\"\n", - " \n", - " # Replace ':' with unit letters, if present.\n", - " if ':' in coord:\n", - " chunks = coord.split(':', 2)\n", - " for n in range(len(chunks)):\n", - " new = \"\".join([new, chunks[n], units[n]])\n", - " else:\n", - " new = coord\n", - " \n", - " # Check that each unit letter is present before returning string.\n", - " for u in units:\n", - " if u not in new:\n", - " raise ValueError(\"Could not parse {}\".format(coord))\n", - " else:\n", - " return new" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Select our coordinates\n", - "RA = '04:16:09.370'\n", - "DEC = '-24:04:20.50'\n", - "SAMPLE_COORDS = check_user_coordinates(RA, DEC)\n", - "print(SAMPLE_COORDS)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that we have our RA and Dec available in degrees, we can define a radius (also in degrees) and submit a filtered position query. Recommended search radii for various JWST observing modes are provided within the JWST Duplication Policy information found on JDox. By default, we keep the \"columns\":\"COUNT_BIG(*)\" to simply return a count of the results, in case we hit a large number of entries.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def filtered_position_query(coordinates, filters):\n", - " \"\"\"\n", - " Construct a filtered position mashup request to send to the mastQuery module. Return \n", - " either the results of the query or just the results count.\n", - " \n", - " Parameters\n", - " ----------\n", - " coordinates : tuple\n", - " Expects a pair of coordinates in degrees.\n", - " filters : dict\n", - " Define the filters to apply to the query.\n", - " \n", - " Returns\n", - " -------\n", - " queryResults : dict\n", - " A dictionary of query results with keys of 'data', 'fields', 'msg',\n", - " 'paging', and 'status'\n", - " \"\"\"\n", - " \n", - " # Make sure the filters dict is not empty.\n", - " assert len(filters.keys()) > 0\n", - " \n", - " # Check the provided coordinates.\n", - " coords = check_user_coordinates(*coordinates)\n", - " \n", - " queryResults = None\n", - " try:\n", - " queryResults = submit_mashup_request(filters, coordinates=coords)\n", - " except RuntimeError as err:\n", - " print(\"QUERY ERROR: {}\".format(err))\n", - " \n", - " return queryResults" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "TEST_QUERY = filtered_position_query(SAMPLE_COORDS, MY_FILTS)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the above result we see we get 152 matches for a search on JWST at this position. We can now use the pandas DataFrame class for a better look at some of the information returned from our query.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd\n", - "\n", - "def convert_to_dataframe(queryResults):\n", - " \"\"\"Create a Pandas DataFrame from a MAST query results 'data' dictionary but just\n", - " return the first few rows.\n", - " \n", - " Parameters\n", - " ----------\n", - " queryResults : dict\n", - " The returned results from a MAST query.\n", - " \n", - " Returns\n", - " -------\n", - " frame : pandas.DataFrame\n", - " The results as a DataFrame object.\n", - " \"\"\"\n", - " \n", - " frame = pd.DataFrame.from_dict(queryResults['data'])\n", - " return frame" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "TEST_FRAME = convert_to_dataframe(TEST_QUERY)\n", - "TEST_FRAME[:4]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can also use our DataFrame to see a list of all the columns returned by the MAST query." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "list(TEST_FRAME.columns)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can use this list to further narrow our query and more closely match our own observations." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "MY_FILTS = {\"obs_collection\": \"JWST\",\n", - " \"instrument_name\": \"NIRISS/WFSS\",\n", - " \"dataproduct_type\": \"spectrum\",\n", - " \"filters\": \"GR150C;F200W\"\n", - " }\n", - "\n", - "TEST_QUERY = filtered_position_query(SAMPLE_COORDS, MY_FILTS)\n", - "convert_to_dataframe(TEST_QUERY)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "Analyzing our results
\n", - "Now that we have some full query results available, we'll want to extract some of the relevant information. First, we'll want to see a list of any nearby pointings found by our position query. Next, we'll also want to see a list of which programs these pointings are associated with.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def analyze_query_results(our_target, queryResults):\n", - " \"\"\"\n", - " Perform analysis of the query results. Identify the nearby planned pointings and\n", - " identify the programs involved.\n", - " \n", - " :param our_target: Expects a pair of coordinates in degrees.\n", - " :type our_target: tuple\n", - " \n", - " :param queryResults: Full results from a MAST query.\n", - " :type queryResults: dictionary\n", - " \"\"\"\n", - " \n", - " # Set up initial variables\n", - " data = queryResults['data']\n", - " ra_target, dec_target = our_target\n", - " targets = {}\n", - " programs = []\n", - " \n", - " # Create a dictionary of all unique coordinate pairs along with a count of how many times they\n", - " # are found\n", - " for current in data:\n", - " current_program = current['proposal_id']\n", - " current_ra = current['s_ra']\n", - " current_dec = current['s_dec']\n", - " current_coords = (current_ra, current_dec)\n", - " if current_coords in targets.keys():\n", - " targets[current_coords] += 1\n", - " else:\n", - " targets[current_coords] = 1\n", - " programs.append(current_program)\n", - "\n", - " # For each unique coordinate pair, calculate the distance from the target and display our \n", - " # results\n", - " for x in sorted(targets.keys()):\n", - " num_obs = targets[x]\n", - " unique_ra, unique_dec = x\n", - " result = \"Found {0} planned observations at {1}, {2}\".format(num_obs, \n", - " unique_ra, \n", - " unique_dec)\n", - " distance_ra = abs(unique_ra - ra_target)\n", - " distance_dec = abs(unique_dec - dec_target)\n", - " distance = SkyCoord(distance_ra, distance_dec, frame=\"icrs\", unit='deg')\n", - " if distance_ra < 0.001 and distance_dec < 0.001: # Account for rounding differences\n", - " result += \" (target match)\"\n", - " else:\n", - " result += \" ({0} away)\".format(distance.to_string('hmsdms'))\n", - " print(result)\n", - " \n", - " # Output a link for each program found\n", - " for p in sorted(list(set(programs))):\n", - " address = \"https://jwst.stsci.edu/observing-programs/program-information?id={0}\".format(p)\n", - " print(\"Found planned observations in {0}: {1}\".format(p, address))\n", - " \n", - " # Return the dictionary of coordinates and number of observations\n", - " return targets" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "TEST_RESULTS = analyze_query_results(SAMPLE_COORDS, TEST_QUERY)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Instead of simply listing associated proposal ID's, our module creates a link to the program information page for any associated proposals found in the area. On these pages, the user may access additional information on the program or the associated APT file itself. Examining the information provided on these program pages is a critical step to investigating any potential sources of conflict for your JWST observations.
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "Saving results to a file
\n", - "This gives us a basic idea of how many observations are currently planned in the vicinity, but this does not necessarily mean these observations will conflict. We still need to check which instruments and configurations are being used. Much of this information is contained in the query results we've obtained, and would be more easily digestable in a CSV table.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import csv\n", - "\n", - "def write_to_csv_file(queryResults, filename):\n", - " \"\"\"\n", - " Write MAST query results to a .csv file.\n", - " \n", - " :param queryResults: Full results from a MAST query.\n", - " :type queryResults: dictionary\n", - " \n", - " :param filename: The desired filename to save the CSV table as.\n", - " :type filename: string\n", - " \"\"\"\n", - " \n", - " # Column names are stored in the 'fields' dictionary\n", - " fields = queryResults['fields']\n", - " header = [entry['name'] for entry in fields]\n", - "\n", - " # Use the DictWriter class to write the data dictionary to a .csv file\n", - " directory = os.getcwd()\n", - " filename = \"/\".join([directory, filename])\n", - " data = queryResults['data']\n", - " with open(filename, 'w') as output:\n", - " writer = csv.DictWriter(output, fieldnames=header)\n", - " writer.writeheader()\n", - " w = [writer.writerow(obs) for obs in data]\n", - " \n", - " # Return the filename created\n", - " print(\"Saved {0}\".format(filename))\n", - " return filename" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Choose a filename for the resulting CSV table\n", - "SAVE_AS = 'planned_obs_test.csv'\n", - "SAVED = write_to_csv_file(TEST_QUERY, SAVE_AS)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "Process multiple targets
\n", - "We now have a CSV table with all parameters available through the Portal for all observations found within a 0.2 degree radius of a set of sample coordinates. The API allows us to now take this one step further and bring all these modules together and check multiple sets of coordinates back-to-back.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def check_multiple_targets(query_filters, coordinates_list, write=False):\n", - " \"\"\"\n", - " Handle the full process for checking a list of targets. This includes converting\n", - " to degrees if necessary, sending an initial count query, sending the full MAST\n", - " query, analyzing the results, and writing the results to a file.\n", - " \n", - " :param coordinates_list: Expects a list of coordinate tuples, not necessarily in\n", - " degrees.\n", - " :type coordinates_list: list\n", - " \n", - " :param coordinates_format: Which format the list of coordinates is provided in\n", - " will determine whether or not convert_to_degrees is run.\n", - " :type coordinates_format: string\n", - " \n", - " :param write: Flag to save results to a .csv file.\n", - " :type write: boolean\n", - " \"\"\"\n", - " \n", - " results = []\n", - " \n", - " # Iterate through our list of coordinate tuples\n", - " for target in coordinates_list:\n", - " query_results = None\n", - " print(\"...checking {0}...\".format(target))\n", - " \n", - " # Convert each pair of coordinates to degrees if necessary\n", - " target = check_user_coordinates(target)\n", - " \n", - " # Submit an initial count query\n", - " try:\n", - " query_results = filtered_position_query(target, query_filters)\n", - " except RuntimeError as err:\n", - " print(err)\n", - " continue\n", - " \n", - " # Analyze the results\n", - " if query_results:\n", - " nearby_targets = analyze_query_results(target, query_results)\n", - " results.append(query_results)\n", - " \n", - " # Generate a filename and write CSV table if 'write' enabled\n", - " if write:\n", - " filename = \"results_{0}_{1}.csv\".format(target[0], target[1])\n", - " filename = write_to_csv_file(query_results, filename)\n", - " \n", - " # Return the list of all query result dictionaries\n", - " return results" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "OUR_COORDINATES_LIST = [('05h42m15s', '+48d22m43s'),\n", - " ('19h45m01.190s', '-14d45m15.79s'),\n", - " ('20h20m20.20s', '+20d20m20.20s'),\n", - " ('04h16m09.370s', '-24d04m20.50s'),\n", - " ]\n", - "OUR_QUERY_RESULTS = check_multiple_targets(MY_FILTS, OUR_COORDINATES_LIST, write=False) # Change the 'write' flag to save .csv files" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "Getting coordinates from stationary target names
\n", - "Using the API also gives us access to the name-resolver service, which allows us to input a list of target names instead of coordinates. The returned coordinates are already in degrees, so we can also skip the convert_to_degrees module.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def resolve_target_names(target_list):\n", - " \"\"\"\n", - " Look up a list of target names using the MAST lookup service and return\n", - " a list of coordinates.\n", - " \n", - " :param target_list: The list of targets to be resolved.\n", - " :type target_list: list\n", - " \"\"\"\n", - " \n", - " # Set up an empty list for coordinate results and iterate through the list of target names\n", - " coordinates_list = []\n", - " for target_name in target_list:\n", - " \n", - " # Make a resolver request with the current target name\n", - " resolverRequest = {'service':'Mast.Name.Lookup',\n", - " 'params':{'input':target_name,\n", - " 'format':'json'}\n", - " }\n", - " headers, resolvedObjectString = mastQuery(resolverRequest)\n", - " resolvedObject = json.loads(resolvedObjectString)\n", - " \n", - " # If the target name was not found, we will run into IndexErrors when we try to set these \n", - " # variables\n", - " try:\n", - " target_ra = resolvedObject['resolvedCoordinate'][0]['ra']\n", - " target_dec = resolvedObject['resolvedCoordinate'][0]['decl']\n", - " canonical_name = resolvedObject['resolvedCoordinate'][0]['canonicalName']\n", - " except IndexError:\n", - " print(\"{0} not found\".format(target_name))\n", - " continue\n", - " \n", - " # Add the coordinates as a tuple to the list\n", - " target_coords = (target_ra, target_dec)\n", - " coordinates_list.append(target_coords)\n", - " print(\"Found {0} at {1}\".format(canonical_name, target_coords))\n", - " \n", - " # Return the list of coordinate tuples\n", - " return coordinates_list" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "OUR_TARGETS_LIST = ['M92',\n", - " '30 Doradus',\n", - " 'Dumbbell Nebula',\n", - " 'Fake Test Galaxy'\n", - " ]\n", - "NEW_COORDINATES_LIST = resolve_target_names(OUR_TARGETS_LIST)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "With our list of names resolved we can send our new list of coordinates to our previous query module." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Change the 'write' flag to save .csv files\n", - "NEW_QUERY_RESULTS = check_multiple_targets(MY_FILTS, NEW_COORDINATES_LIST, write=False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "
Searching for moving targets (keyword searches on target_name)
\n", - "Moving targets are a bit trickier since we do not have a single set of coordinates to search against. Instead, we'll look for matching entries in the target_name field and we'll use the freeText paramter with wildcards to find any close matches.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def filtered_keyword_query(query_filters, keyword):\n", - " \"\"\"\n", - " Construct a filtered mashup request to send to the mastQuery function. This\n", - " will search for a given keyword within the 'target_name' field and return\n", - " any results including wildcards.\n", - " \n", - " :param keyword: The keyword to search for within 'target_name'.\n", - " :type keyword: string\n", - " \n", - " :param count: Flag to submit a full query or a count query. Sends count\n", - " by default.\n", - " :type count: boolean\n", - " \"\"\"\n", - " \n", - " queryResults = None\n", - " query_filters[\"target_name\"] = keyword\n", - " try:\n", - " queryResults = submit_mashup_request(query_filters)\n", - " except RuntimeError as err:\n", - " print(err)\n", - " \n", - " return queryResults" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "MY_FILTS = {\"obs_collection\": \"JWST\"}\n", - "\n", - "JUPITER_QUERY = filtered_keyword_query(MY_FILTS, \"Jupiter\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "With query results in hand we'll build a module to analyze our moving target output. We'll want lists of the target names we matched with and which programs these are found in.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def keyword_matches(queryResults):\n", - " \"\"\"\n", - " Create a list of the unique target names that were returned by the filtered_keyword_query.\n", - " Also generate a link to each nearby program found.\n", - " \n", - " :param queryResults: Full results from a MAST query.\n", - " :type queryResults: dictionary\n", - " \"\"\"\n", - " \n", - " # Set up initial variables\n", - " data = queryResults['data']\n", - " targets = []\n", - " programs = []\n", - " \n", - " # For each query result, pull out the target_name and proposal_id values\n", - " for obs in data:\n", - " current_target = obs['target_name']\n", - " current_program = obs['proposal_id']\n", - " targets.append(current_target)\n", - " programs.append(current_program)\n", - " \n", - " # Get the unique sets of target_name and proposal_id values\n", - " unique_targets = sorted(list(set(targets)))\n", - " unique_programs = sorted(list(set(programs)))\n", - " \n", - " # Output our results\n", - " print(\"Matched the keyword to: {0}\".format(unique_targets))\n", - " for p in unique_programs:\n", - " address = \"https://jwst.stsci.edu/observing-programs/program-information?id={0}\".format(p)\n", - " print(\"Found planned observations in {0}: {1}\".format(p, address))\n", - " \n", - " # Return the list of unique target names\n", - " return unique_targets" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "JUPITER_MATCHES = keyword_matches(JUPITER_QUERY)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Copying our multiple targets module from above, with a few tweaks we can now run back-to-back queries on a list of moving target names. We can also incorporate the write_to_csv_file module to automatically save all our results for further inspection.
" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def check_multiple_keywords(query_filters, keywords_list, write=False):\n", - " \"\"\"\n", - " Perform multiple filtered MAST queries based on a list of keywords to look\n", - " for within the 'target_name' field. Analyze the results and write them to\n", - " a .csv file if requested.\n", - " \n", - " :param keywords_list: List of target keywords to search for in the 'target_name'\n", - " field.\n", - " :type keywords_list: list\n", - " \n", - " :param write: Flag whether or not the results are save in a .csv file.\n", - " :type write: boolean\n", - " \"\"\"\n", - " \n", - " results = []\n", - " \n", - " # Iterate through our list of target_name keywords\n", - " for target in keywords_list:\n", - " query_results = None\n", - " print(\"...checking {0}...\".format(target))\n", - " \n", - " # Submit an initial count query\n", - " query_results = filtered_keyword_query(query_filters, target)\n", - " \n", - " # If the count is within a valid range, submit the full query\n", - " if query_results:\n", - " targ_names = keyword_matches(query_results)\n", - " results.append(query_results)\n", - " \n", - " # Generate a filename and write CSV table if 'write' enabled\n", - " if write:\n", - " filename = \"results_{0}.csv\".format(target)\n", - " filename = write_to_csv_file(query_results, filename)\n", - " \n", - " # Return the list of query result dictionaries\n", - " return results" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "OUR_MOVING_TARGETS = [\"Jupiter\",\n", - " \"Titan\",\n", - " \"Ceres\",\n", - " \"Sun\"\n", - " ]\n", - "# Change the 'write' flag to save .csv files\n", - "MOVING_MATCHES = check_multiple_keywords(MY_FILTS, OUR_MOVING_TARGETS)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## About this Notebook\n", - "\n", - "This notebook was developed by Archive Sciences Branch staff, chiefly Peter Forshay and Dick Shaw. Additional editing was provided by Thomas Dutkiewicz.\n", - "\n", - "For support, please contact the Archive HelpDesk at archive@stsci.edu.\n", - "\n", - "Last updated: August 2023\n", - "\n", - ""
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/notebooks/JWST/duplication_checking/targets.csv b/notebooks/JWST/duplication_checking/targets.csv
deleted file mode 100644
index dd6e5fc6..00000000
--- a/notebooks/JWST/duplication_checking/targets.csv
+++ /dev/null
@@ -1,6 +0,0 @@
-target_name, RA, DEC
-CX Tau, 4:14:47.861, +26:48:10.91
-Fomalhaut, 22:57:39.046, -29:37:20.05
-HL Tau, 0:31:38.437, +18:13:57.65
-M 8, 18:03:36.960, -24:23:13.20
-30 Dor, 5:38:42.396, -69:06:03.36
\ No newline at end of file