-
Notifications
You must be signed in to change notification settings - Fork 10
Creating Legacy lvl dbd Structures
kerfoot edited this page Mar 25, 2014
·
6 revisions
This recipe covers how to export the dataset contained in a DbdGroup to a legacy level 0, 1 or 2 dbd data structure. The dbd data structure is the primary data structure used for automated real-time processing of RU-COOL glider fleet deployments. We will be transitioning to the SPT toolbox for future data management operations but I wanted to provide this example as I recognize that may be more comfortable working with this legacy data structure.
Let's outline the steps involved and then we'll walk through a real-world example:
- Get the list of native glider data files.
- Create an instance of the DbdGroup class. 3.Set the instance timestamp ( DbdGroup.timestampSensors ) and depth ( DbdGroup.depthSensors ) properties.
- Optionally, remove any non-essential sensors.
- Export the dataset to a dbd structure.
You can also see my default usage for the dbds2DbdGroup function to create a new DbdGroup instance here.
###Get the File Listing###
>> pwd
ans =
/home/kerfoot/sandbox/glider/deployments-test/2013/ru07-383/ascii/debd
>> dbds = listDbds('ext', '.dat');
>> size(dbds)
ans =
684 1
###Create the DbdGroup Instance###
Create an instance of the DbdGroup class by feeding the dbd file list to the dbds2DbdGroup.m function:
>> dgroup = dbds2DbdGroup(dbds)
>> dgroup
dgroup =
DbdGroup with properties:
segments: {684x1 cell}
sourceFiles: {684x1 cell}
bytes: [684x1 double]
rows: [684x1 double]
timestampSensors: {684x1 cell}
depthSensors: {684x1 cell}
startTimes: {684x1 cell}
endTimes: {684x1 cell}
startDatenums: [684x1 double]
endDatenums: [684x1 double]
sensors: {69x1 cell}
sensorUnits: [1x1 struct]
numProfiles: [684x1 double]
dbds: [1x684 Dbd]
hasBoundingGps: [684x1 logical]
newSegments: {684x1 cell}
scratch: []
With no options specified, the following processing steps are performed by dbds2DbdGroup.m:
- An instance of the Dbd class, containing all sensors in the raw data file, is created for each file. Default timestamp and depth/pressure sensors are chosen from those available in Dbd.dbdTimestampSensors and Dbd.dbdDepthSensors, respectively, with no interpolation.
- The Dbd instance is added to a newly created instance of the DbdGroup class.
Once all files have been processed and added to the DbdGroup, the following steps are performed:
- Derived CTD sensors ( drv_sea_water_temperature, drv_sea_water_electrical_conductivity, drv_sea_water_salinity, drv_sea_water_density, drv_sea_water_potential_temperature and drv_speed_of_sound_in_sea_water ) are added.
- m_gps_lat and m_gps_lon, if present, are converted from decimal minutes to decimal degrees and added as new sensors drv_m_gps_lat and drv_m_gps_lon, respectively. Default values for each sensor (69696969) are removed from the drv* sensors, but left in the original sensor arrays.
- The distance along track is calculated and added as the sensor drv_distance_along_track.
There are many options available for dbds2DbdGroup.m:
>> help dbds2DbdGroup
dgroup = dbds2DbdGroup(dbd_list[,DbdGroup, varargin])
Creates a new DbdGroup instance containing the Dbd instances created from
the files contained in dbd_list. Default timestamp and depth sensors are
used. The following measured/derived CTD sensors are added to the instance:
'drv_sea_water_temperature'
'drv_sea_water_electrical_conductivity'
'drv_sea_water_salinity'
'drv_sea_water_density'
'drv_sea_water_potential_temperature'
'drv_speed_of_sound_in_sea_water'
If a valid DbdGroup instance is specified as an optional second argument,
the Dbd instances are added to this DbdGroup.
Options must be specified as name, value pairs:
'sensors', [CELL ARRAY]
List of sensor names to include in each Dbd instance.
'timestampsensor', [STRING or CELL ARRAY]
Sensor name to be used as the timestamp sensor, Dbd.timestampSensor,
for each Dbd instance, if available. If not available, a default
sensor is chosen automatically.
'depthsensor', [STRING or CELL ARRAY]
Sensor name to be used as the depth sensor, Dbd.depthSensor, for each
Dbd instance, if available. If not available, a default sensor is
chosen automatically.
'convertgps', [LOGICAL]
Set to true to convert m_gps_lat and m_gps_lon to decimal degrees and
add as new sensors drv_m_gps_lat and drv_m_gps_lon. Default is true.
'replace', [LOGICAL]
Set to true to replace existing Dbd instances for which the size of
the source data file has changed. Default is true.
'fillgps', [LOGICAL]
Set to true to linearly interpolate measured gps fixes and store the
values ('drv_longitude' and 'drv_latitude'). Interpolation is
only performed on segments in which the segment is bound by a pre-dive
and post-dive gps fix, which is specified by the 'hasBoundingGps'
property of the Dbd instance. Default is true.
'deletesource', [LOGICAL]
Set to true to delete the source data file after processing,
regardless of whether a Dbd instance was added to the DbdGroup
instance. Default is false.
'addctdsensors', [LOGICAL]
Set to false to prevent the addition of derived CTD parameters.
'promintimespan', [NUMBER]
Number of seconds an indexed profile must span to be considered a
profile. The default value is set when an instance of the Dbd
class is created.
'promindepthspan', [NUMBER]
Number of meters an indexed profile must span to be considered a
profile. The default value is set when an instance of the Dbd class
is created.
'promindepth', [NUMBER]
The minumum depth, in meters, that may be used when indexing profiles.
The default value is set when an instance of the Dbd class
is created.
'prominnumpoints', [NUMBER]
Minimum number of points that an indexed profile must contain to be
considered a profile. The default value is set when an instance of
the Dbd class is created.
###Set Instance Timestamp & Depth Sensor###
This DbdGroup instance contains one Dbd instance for each successfully processed native glider data file. Default timestamp & depth sensors are chosen for each Dbd instance, depending on the available sensors. Since we're typically using these data structures for analyzing the scientific data sets, we'll set the timestamp and depth sensor for each Dbd instance manually:
dgroup.timestampSensors = 'drv_sci_m_present_time_datenum'
dgroup.depthSensors = 'drv_sci_water_pressure'
###Recommended DbdGroup Creation Usage###
I typically use the following options with dbds2DbdGroup.m when creating a new DbdGroup:
>> dgroup = dbds2DbdGroup(dbds, 'timestampsensor', 'drv_sci_m_present_time', 'depthsensor', 'drv_sci_water_pressure')
###Remove Non-Essential Sensors###
In order to decrease the size of the data structure, I usually remove all non-essential sensors prior to saving the scientific data structure. This can be done using a cell array of regular expressions (located in /constants) as an input to the DbdGroup.deleteSensors('regexp', CELL_ARRAY) method. Here's the current list of sensors in the DbdGroup instance:
>> dgroup.sensors
ans =
'c_fin'
'c_heading'
'c_pitch'
'c_roll'
'c_wpt_lat'
'c_wpt_lon'
'drv_distance_along_track'
'drv_latitude'
'drv_longitude'
'drv_m_gps_lat'
'drv_m_gps_lon'
'drv_m_present_time_datenum'
'drv_m_pressure'
'drv_proDir'
'drv_proInds'
'drv_sci_ctd41cp_timestamp_datenum'
'drv_sci_m_present_time_datenum'
'drv_sci_water_pressure'
'drv_sea_water_density'
'drv_sea_water_electrical_conductivity'
'drv_sea_water_potential_temperature'
'drv_sea_water_salinity'
'drv_sea_water_temperature'
'drv_speed_of_sound_in_seawater'
'm_altitude'
'm_appear_to_be_at_surface'
'm_coulomb_amphr'
'm_depth'
'm_fin'
'm_final_water_vx'
'm_final_water_vy'
'm_gps_full_status'
'm_gps_lat'
'm_gps_lon'
'm_gps_status'
'm_heading'
'm_iridium_call_num'
'm_iridium_redials'
'm_iridium_signal_strength'
'm_lat'
'm_lon'
'm_pitch'
'm_present_time'
'm_pressure'
'm_roll'
'm_science_clothesline_lag'
'm_vacuum'
'm_water_depth'
'm_water_vx'
'm_water_vy'
'sci_bb3slo_b470_ref'
'sci_bb3slo_b470_scaled'
'sci_bb3slo_b470_sig'
'sci_bb3slo_b532_ref'
'sci_bb3slo_b532_scaled'
'sci_bb3slo_b532_sig'
'sci_bb3slo_b660_ref'
'sci_bb3slo_b660_scaled'
'sci_bb3slo_b660_sig'
'sci_bb3slo_is_installed'
'sci_bb3slo_temp'
'sci_bb3slo_timestamp'
'sci_bbfl2s_bb_ref'
'sci_bbfl2s_bb_scaled'
'sci_bbfl2s_bb_sig'
'sci_bbfl2s_cdom_ref'
'sci_bbfl2s_cdom_scaled'
'sci_bbfl2s_cdom_sig'
'sci_bbfl2s_chlor_ref'
'sci_bbfl2s_chlor_scaled'
'sci_bbfl2s_chlor_sig'
'sci_bbfl2s_temp'
'sci_bbfl2s_timestamp'
'sci_ctd41cp_timestamp'
'sci_m_present_time'
'sci_water_cond'
'sci_water_pressure'
'sci_water_temp'
'x_software_ver'
>> length(dgroup.sensors)
ans =
79
Let's remove the sensors we don't need:
>> SCI_EXCLUDE_SENSORS = flightControllerSensorRegexps()
SCI_EXCLUDE_SENSORS =
'^c_'
'^cc_'
'^dc_'
'^dhs_'
'^f_'
'^s_'
'^u_'
'^xs_'
'^x_'
'm_[xy]_lmc'
'm_veh_temp'
'm_leakdetect_'
'm_heading'
'm_hdg_error'
'm_gps_status'
'm_gps_mag_var'
'm_fin$'
'm_battpos'
'm_ballast_pumped'
>> dgroup.deleteSensor('regexp', SCI_EXCLUDE_SENSORS)
>> dgroup.sensors
ans =
'drv_distance_along_track'
'drv_latitude'
'drv_longitude'
'drv_m_gps_lat'
'drv_m_gps_lon'
'drv_m_present_time_datenum'
'drv_m_pressure'
'drv_proDir'
'drv_proInds'
'drv_sci_ctd41cp_timestamp_datenum'
'drv_sci_m_present_time_datenum'
'drv_sci_water_pressure'
'drv_sea_water_density'
'drv_sea_water_electrical_conductivity'
'drv_sea_water_potential_temperature'
'drv_sea_water_salinity'
'drv_sea_water_temperature'
'drv_speed_of_sound_in_seawater'
'm_altitude'
'm_appear_to_be_at_surface'
'm_coulomb_amphr'
'm_depth'
'm_final_water_vx'
'm_final_water_vy'
'm_gps_full_status'
'm_gps_lat'
'm_gps_lon'
'm_iridium_call_num'
'm_iridium_redials'
'm_iridium_signal_strength'
'm_lat'
'm_lon'
'm_pitch'
'm_present_time'
'm_pressure'
'm_roll'
'm_science_clothesline_lag'
'm_vacuum'
'm_water_depth'
'm_water_vx'
'm_water_vy'
'sci_bb3slo_b470_ref'
'sci_bb3slo_b470_scaled'
'sci_bb3slo_b470_sig'
'sci_bb3slo_b532_ref'
'sci_bb3slo_b532_scaled'
'sci_bb3slo_b532_sig'
'sci_bb3slo_b660_ref'
'sci_bb3slo_b660_scaled'
'sci_bb3slo_b660_sig'
'sci_bb3slo_is_installed'
'sci_bb3slo_temp'
'sci_bb3slo_timestamp'
'sci_bbfl2s_bb_ref'
'sci_bbfl2s_bb_scaled'
'sci_bbfl2s_bb_sig'
'sci_bbfl2s_cdom_ref'
'sci_bbfl2s_cdom_scaled'
'sci_bbfl2s_cdom_sig'
'sci_bbfl2s_chlor_ref'
'sci_bbfl2s_chlor_scaled'
'sci_bbfl2s_chlor_sig'
'sci_bbfl2s_temp'
'sci_bbfl2s_timestamp'
'sci_ctd41cp_timestamp'
'sci_m_present_time'
'sci_water_cond'
'sci_water_pressure'
'sci_water_temp'
>> length(dgroup.sensors)
ans =
69
###Export the Dataset###
Use the DbdGroup class method toDbdStruct to export the data set as a lvl0,1 or 2 data structure:
>> length(dgroup.sensors)
ans =
69
>> lvl = dgroup.toDbdStruct()
lvl =
depth: 2
drv_distance_along_track: 3
drv_latitude: 4
drv_longitude: 5
drv_m_gps_lat: 6
drv_m_gps_lon: 7
drv_m_present_time_datenum: 8
drv_m_pressure: 9
drv_proDir: 10
drv_proInds: 11
drv_sci_ctd41cp_timestamp_datenum: 12
drv_sci_m_present_time_datenum: 13
drv_sci_water_pressure: 14
drv_sea_water_density: 15
drv_sea_water_electrical_conductivity: 16
drv_sea_water_potential_temperature: 17
drv_sea_water_salinity: 18
drv_sea_water_temperature: 19
drv_speed_of_sound_in_seawater: 20
m_altitude: 21
m_appear_to_be_at_surface: 22
m_coulomb_amphr: 23
m_depth: 24
m_final_water_vx: 25
m_final_water_vy: 26
m_gps_full_status: 27
m_gps_lat: 28
m_gps_lon: 29
m_iridium_call_num: 30
m_iridium_redials: 31
m_iridium_signal_strength: 32
m_lat: 33
m_lon: 34
m_pitch: 35
m_present_time: 36
m_pressure: 37
m_roll: 38
m_science_clothesline_lag: 39
m_vacuum: 40
m_water_depth: 41
m_water_vx: 42
m_water_vy: 43
sci_bb3slo_b470_ref: 44
sci_bb3slo_b470_scaled: 45
sci_bb3slo_b470_sig: 46
sci_bb3slo_b532_ref: 47
sci_bb3slo_b532_scaled: 48
sci_bb3slo_b532_sig: 49
sci_bb3slo_b660_ref: 50
sci_bb3slo_b660_scaled: 51
sci_bb3slo_b660_sig: 52
sci_bb3slo_is_installed: 53
sci_bb3slo_temp: 54
sci_bb3slo_timestamp: 55
sci_bbfl2s_bb_ref: 56
sci_bbfl2s_bb_scaled: 57
sci_bbfl2s_bb_sig: 58
sci_bbfl2s_cdom_ref: 59
sci_bbfl2s_cdom_scaled: 60
sci_bbfl2s_cdom_sig: 61
sci_bbfl2s_chlor_ref: 62
sci_bbfl2s_chlor_scaled: 63
sci_bbfl2s_chlor_sig: 64
sci_bbfl2s_temp: 65
sci_bbfl2s_timestamp: 66
sci_ctd41cp_timestamp: 67
sci_m_present_time: 68
sci_water_cond: 69
sci_water_pressure: 70
sci_water_temp: 71
timestamp: 1
data: [1738911x71 double]
source: {684x7 cell}
meta: [1x1 struct]
Save the resulting data structure:
>> save('ru07-383_archive_lvl2.mat', 'lvl')