Fix the NSF award API endpoint

[jeanetteclark]

Describe the bug

Currently, for the NSF award lookup on the ADC, metacatUI looks at https://api.nsf.gov/services/v1/awards.json. This endpoint has a redirect that doesn't work so well and was causing failing award lookups in metadig engine. I changed the proxy in our apache config to research.gov/awardapi-service/v1/awards.json. I think that at a minimum the metacatUI lookup should move to that endpoint, and probably it should be using the proxy instead. @mbjones indicated that this might be overridden in a config somewhere so possibly a non issue, but the award lookup doesn't seem to be working for me right now.

[mbjones]

Thanks for the report, @jeanetteclark. The reason that we used the proxy in the first place was to bypass Same Origin restrictions on 3rd party API access. I'm not sure what changed there, maybe we fixed it through CORS headers? In any case, @robyngit and @rushirajnenuji, this is a problem that needs to get fixed ASAP, as it is hitting the datateam members. Could one of you look into it -- hopefully it will be a minor issue. Thanks.

[jeanetteclark]

FYI I switched the apache config back. The award lookup is still not working in metacatUI even with the old config. I also checked with the current datateam and they say the UI lookup has not worked for a while now.

[robyngit]

Notes:

• The default URL is set in the App model here.
• new API endpoint: "https://www.research.gov/awardapi-service/v1/awards.json?id="
• old API endpoint: "https://api.nsf.gov/services/v1/awards.json?id="

[mbjones]

Another report of problems with the award lookup from RT today:

It looks like I need to provide overview info on the dataset and then link it with my NSF grant. However, when I type in the award number (or anything) in the funding box, nothing emerges from the search.

[robyngit]

Updating the ADC config to use the new endpoint for the grantsUrl did not resolve this NSF award lookup issue. The URLs formed by MetacatUI were valid, in that they returned a 200 status, but the dropdown in the UI still did not appear. In the browser, the URLs consistently gave 0 results. Updating the grantsUrl to remove the ?id= suffix from the URL did allow the endpoint to return the expected results, but the dropdown in the UI still did not appear. This suggests that the issue extends beyond just the endpoint URL configuration, and that there are deeper issues in the UI component handling or data processing within MetacatUI. Further investigation is needed to resolve the UI dropdown issue fully.

The grantsUrl setting that formed valid URLs was: "https://www.research.gov/awardapi-service/v1/awards.json?id="

The search term "arctica" should return results.

Here is an example query that works in the browser: https://www.research.gov/awardapi-service/v1/awards.json?keyword=arctica&printFields=title,id&callback=jQuery191024293065478125508_1704926101036&_=1704926101037

[robyngit]

Looking further into this, the request to the awards API is failing because of a CORS issue: the response is blocked by the same-origin policy. It appears that we were previously getting around this by requesting the API to return JSONP (JSONP is not subject to the same-origin policy). Although "callback" is still a valid parameter accepted by the API, it looks like the response is no longer being wrapped in a callback function. I tested this both in the browser and with curl, and the response is plain JSON, not JSONP. I also tried the same request with the old endpoint, and using the "jsonp" parameter instead of "callback"; neither returned JSONP.

The only documentation that I can find for this API appears to be out of date since it's still referencing the old endpoint. I'm not sure if there's any documentation for the new endpoint, but it would be helpful to know if JSONP is still supported.

Possible solutions:

1. If JSONP is still supported, we can update our code to request JSONP from the API however it's currently done.
2. Set up a proxy on our servers to handle the CORS issue.
3. Contact NSF to see if they can provide a solution or update the CORS policy on their end.

Update: we do in fact already have a proxy set up for this API, it's just not working as expected, investigating this now...

[robyngit]

Here is the history of this issue (found by @mbjones 🔍 ): #629

[robyngit]

After much slack discussion, we've got the problem narrowed down:

Historically MetacatUI was using JSONP to get around NSF's lack of CORS support. NSF did a rewrite of their API (hence the change in the endpoint URL) and it seems that they dropped support for JSONP at the same time. Given the lack of support for CORS and JSONP, MetacatUI will need to make requests via a server-side proxy.

The Apache setup for the ADC was already configured to proxy requests to the NSF award API, however, the proxy needed to be updated to the correct URL (including www and using https instead of http). This has now been corrected, so we can now update MetacatUI to make requests to the proxy and to request and handle JSON responses rather than JSONP.

For the ADC, the grantsUrl that we should use to lookup grants via the proxy is: https://arcticdata.io/research.gov/awardapi-service/v1/awards.json.

Any repository that wants to use the NSF award lookup feature will need to set up their own proxy. We could configure the default grantsUrl to point to a proxy relative to the repository’s own domain. We will need to add documentation to MetacatUI docs about how to set up a proxy for the NSF award API (and how to override the grantsUrl if it differs from the default)

To Do:

• Update the default grantsUrl to point to a proxy relative to the repository’s domain
• Update MetacatUI to handle JSON responses rather than JSONP
• Add documentation to MetacatUI docs about how to set up a proxy for the NSF award API
• Write tests for these methods