Mirroring
This page is a work in progress and is based on the discussion going on here. The mechanism described below is not implemented yet.
In order to ensure data consistency across the current Helioviewer data providers we have implemented a mirroring scheme to sync data across helioviewer instances. This ensures that any reprocessed data will not stay out of sync across mirrors for long.
The mirroring scheme is intended to be uni-directional, meaning there is always a primary server for a specific data source. However, one server may be the primary server for one observatory, and a mirror server for others.
For example, the ROB server may be the primary server for GONG and Solar Orbiter data while the GSFC server is the primary source for SOHO and STEREO.
A mirror itself may also act as a primary server for other mirrors.
The primary server is responsible for getting processing data from the upstream data providers. The upstream providers process the FITS data collected from an observatory into jpeg2000 files suitable for Helioviewer.
The helioviewer mirror servers get their data from a primary server (or a server that is already mirroring a primary server). This is accomplished through the mirroring mechanism described below.
A Helioviewer server acting as a primary source will host a HAPI server to deliver metadata about new files.
- The datasets on the primary server are TBD (observatory? instrument? measurements?)
- Within each dataset, the following parameters are provided:
| Parameter | Description |
|---|---|
| Time | The time the file was ingested on the local server. This is used to query the list of new files added to the server |
| name | Name of the jp2 file |
| checksum | A checksum to compare with existing local files to see if they are different. |
| url | An http(s) url to the jp2 file |
The mirror client will follow the following flow diagram:
The steps are:
- Query the primary server for the selected data source
- For each file in the response
- If the file is new, ingest it.
- If the file already exists, compare the checksum with the local file.
- If the checksum doesn't match, ingest the file.