Extreme events/anomaly detection package

[sophieclayton]

Anomaly detection to identify extreme events in ocean datasets

Summary

In recent years there's been a growing awareness of extreme or anomalous events in the ocean. Ocean heatwaves are a great example on the physical side, but there are also other types of extreme, ephemeral events that might be good to know about, e.g. hypoxic events, low pH... This project would aim to produce a package that could be used to identify extreme events (physical, chemical, biological) from freely available ocean data.

Personnel

TBD

Specific tasks

• Implement one (or more) anomaly detection algorithms time series (e.g. time series from a mooring?) and/or spatial (e.g. remote sensing) data.
• Develop pipeline for feeding in data from an ocean data stream to the algorithm
• Share the package with the community! (so it'll need a README and a worked example, maybe even a wiki)

Potential data sets to use

• NOAA Ocean Acidification Buoy data
• OOI mooring data

Existing methods and application example

Here is an example of a python package implemented to detect ocean heatwaves using a specific algorithm for ocean heat waves.

My suggestion would be to implement something similar to look at a biogeochemical property, e.g. dissolved oxygen, pH, pCO2, or even Chl.

[cathmmitchell]

This dataset could be useful (details about it here). Luke Gregor presented at the Ocean Carbon From Space Workshop using this dataset to look at ocean acidification extremes and marine heatwaves. His presentation is available on the workshop website.

[abkfenris]

@adamkemberling might be interested in this.

[ocefpaf]

I would say yes but I'm not familiar with heat waves literature to know the details. From a simple anomaly point of view, @emiliom already said, qartod (implemented in ioos_qc), could be helpful. However, most of ioos_qc operations are done in time-series at a single point (some for moving platforms). I wonder if there are fancier grid approaches to solve this.

BTW, didn't we have folks working on this during OHW20/21? We should dig that project up.

[ocefpaf]

Awesome. I recently did some anomaly detection on simple mooring data. Would be nice to build something that could ingest xarray data and look for anomalies in 2-D or 3-D time-varying fields. It could be applied to remote sensing or model output or even argo data.

This package looks promising: https://github.com/coecms/xmhw

[emiliom]

BTW, didn't we have folks working on this during OHW20/21? We should dig that project up.

You're probably referring to OHW20 "Marine Heat Wave (MHW) analysis with xarray", https://oceanhackweek.github.io/ohw21/projects_2020.html

[alessioarena]

I have written a xarray/dask function to predict extreme weather events using Generalised Pareto Distribution on timeseries data. This problem is slightly different but there could be some overlap. Happy to contribute

[MathewBiddle]

@ocefpaf covered any comments I would make. I also wonder if the previous OHW project to match templates to vertical profiles could be leveraged here? Instead of depth, run across x/y?

[emiliom]

This project emphasizes an anomaly as an interesting data cluster. A different but related perspective is that of anomalies as suspicious data clusters in a -- typically sensor based -- data stream. You may find some useful implementations in the QARTOD code, https://ioos.github.io/ioos_qc/

[sophieclayton]

yes, I was thinking about the overlap with QARTOD for this project. Might be interesting to think about how to differentiate between "bad" and "interesting" data beyond what QARTOD already is set up to do

[quachhungvietnam]

Hi Sophie. This is such an interesting idea. If you are looking for someone to collaborate on this project, I am happy to work on this project with you. I already have a cleaned dataset for temperature/salinity/oxygen/... from observations in the northwest Atlantic (can be further customized per interest). These observations can be used for training/testing the model.

[adamkemberling]

I've used this web tool in previous code sprints as way to whiteboard collaboratively:

Putting a shared link here if it is a tool people may want to use this to facilitate discussions
https://app.tryeraser.com/workspace/xfWsnBWyu7piLkR2gsdg?origin=share

[adamkemberling]

A researcher that worked on tracking MHW here went on to do her PhD on the topic: there might be some useful/relevant code here: https://github.com/ocetrac

[adamkemberling]

Dr. Hillary Scannell*

[Maverick2m]

Hi! everyone. I would like to sign up for this project. I am a Master's student working upon the Marine Heatwaves in Indian Ocean. This project definitely falls in my work domain.

[drobinsonn23]

I would be interested in working on this :) It would be interesting to see this package to examine anomalies in oxygen minimum zone intensities and other seasonal biogeochemical properties.

I am a PhD student working on OMZ in Southern California in the field of geomicrobiology, geochemistry and ocean biogeochemistry

[quachhungvietnam]

Hi folks.
I just want to summarize the discussion that we have today on this project and set the direction for this project as our time is short.

1. Choose the variable for this project. Options we have so far: (sea-surface) temperature, productivity/chlorophyll, oxygen. I vote to work for sea-surface temperature because the other two variables (productivity, oxygen)seem unfeasible to do in 1 week. For oxygen, legit oxygen prediction/detection is currently done by tunning earth system model and, the oxygen trend from observation (at least from my research in the Northwest Atlantic) is very inconsistent. For productivity, I highly concern the calibration for satellite source and the in situ data is very limited (at least for Northwest Atlantic). In contrast, data for (sea-surface) temperature is widely available from satellite source (@adamkemberling ) or from in situ measurements, and the temporal trend of (sea-surface) temperature should be predictable (I guess/predict it looks like the CO2 Keeling Curve).
2. Choose the geographical region for our analysis. I'm familiar with the Northwest Atlantic, including the current trends (Gulf Stream, Labrador Current). I have a bias for Gulf of Maine, as I'm in GMRI :), but I can easily request data for other regions if you have a strong opinion on this :)
3. If you agree and want to work on the (sea-surface) temperature prediction with me, two main parallel tasks that I have in my mind and will start working from tomorrow:
   a. Task 1: Request/Maintain temperature data for region (i.e., Gulf of Maine).
   b. Task 2: (big task) Find a framework to predict temperature trend (from data of task 1), then apply that prediction to find the anomaly -> unusual heat wave.

Please let me know if you want to work with me on this project/framework. Otherwise, best of luck with your project!
I desperately hope we can be in the same room to do all these brainstormings together :(

[drobinsonn23]

Hello,

1. I'm okay for using sea surface temperature as a variable and framework for this
2. I'm familiar California and the US West Coast, but the Atlantic region would be fine and it may be best so that this project doesn't seem similar to the ENSO project which may incorporate Pacific SST data into their project. Not sure just speculating
3. • Is there a data repository you know of that includes the Northwest Atlantic? Would NOAA Atlantic Oceanographic Meteorological Laboratory have this data?
   • An idea I have is to find the average sea-surface temperature and graph the anomalies. Then use a forecasting model to predict the anomalies based on previous data. Another idea is to correlate SST with another parameter (current, irradiance ect) then create an equation that can be used to reproduce anomalies (just throwing ideas)
   • Would it be possible to examine how SST influences bottom water temperature anomalies since downwelling and mesoscale eddies can transport surface properties to the ocean interior and deep

Looking forward to this! See you all tomorrow! Also is there a project channel for this?
-De'Marcus

[quachhungvietnam]

Hi Marcus,

The in situ data portal that I request data have worldwide data and include observations from NOAA.
I agree and like your first framework of identifying/the anomalies for SST. I prefer the first one rather than the second one due to the uncertainty of other parameters.
For your suggestion about eddies, if we focus on the shelf water instead of open water, we can mitigate/ignore the eddies impact. For your suggestion about downwelling, I would suggest keeping it as an extension and not the main focus as I am sure if we have sufficient time to collect wind data to model downwelling/upwelling movement.

[adamkemberling]

Hi everyone, lots of good ideas and input during our quick breakout sessions yesterday. I just want to circle back on some of the ideas mentioned yesterday.

The two SST data resources I referred two were:

1. Extended Reconstructed SST (ERSST) which is monthly at 0.5 x 0.5 degree grid https://www.ncei.noaa.gov/products/extended-reconstructed-sst
2. Optimally Interpolated SST (OISSTv2) which has daily records from Fall 1981-Present at 0.25*0.25 degree grid https://www.ncei.noaa.gov/products/optimum-interpolation-sst

I would recommend the latter for accurracy and coverage reasons (newer satellite, higher res, daily data). But I think it will be important for the group to pick a location to study to reduce the amount of data we're iterating over. (Gulf of Mexico and the Loop Current could be cool!)

There also is an opportunity to use sea ice data as well if people wanted to branch out: https://nsidc.org/data/nsidc-0079/versions/3

This morning I will make a slack channel for the project, please message me if an invite hasn't arrived, I will try to grab everyone.

As we start to dig into any of these datasets it might be worth thinking about who wants to tackle different aspects of the project. This can be based on what is most interesting to each of us, what we feel we can contribute to most, or a skill we'd like to work on with the support of our peers.

Potential responsibilities might be:

1. Preparing the regional subset of the dataset used for testing
2. Finding existing resources, testing limitations
3. Documentation!
4. How to approach determining different "extreme events" (is it based on percentile, is it a modeled relationship, what is the reference period, is it an absolute threshold like hypoxia, is a biological reference point)
5. Integrating the event detection function(s) in xarray
6. Event tracking (how to store information about events, event numbers, duration, intensity)
7. Classifying events in space (events among adjacent spatial cells)
8. Package building

Just remember, this project is an opportunity to work together and learn from one another. There is no expectation that the project be a complete and polished project by the end of the week. I think there's a lot of good input so far, and I'm excited to see where the group goes with it!

[drobinsonn23]

These are some great ideas and responsibilities to tackle. I can post some papers on various techniques/methods to quantify extreme events. I'm not experience with making a package from what I read its suppose to be combination of multiple modules and scripts?

[7yl4r]

Hello everyone! The IMaRS lab is doing work that is very closely related to this project. Is there documentation on the final outcome of this project? We're very interested in collaborating on continued development in this area.

[quachhungvietnam]

Hi Tylar,
This is the presentation of our final product: https://www.youtube.com/watch?v=M7gwjNd0uOI&t=1s
Please let me know if you have any further questions/requests.