DBAMERGE Merge data from combined navigation and science data sets into a single data set.
Syntax:
[META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI)
[META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPTIONS)
[META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPT1, VAL1, ...)
Description:
[META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI) merges the
navigation and science data sets described by metadata structs META_NAV and
META_SCI, and data arrays DATA_NAV and DATA_SCI into a single data set
described by metadata struct META and data array or struct DATA
(see format option described below). Input metadata and data should be
in the format returned by the function DBACAT. Sensor cycles from both
data sets are merged based on the order of the respective timestamps.
See note on merging process.
[META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPTIONS) and
[META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPT1, VAL1, ...)
accept the following options given in key-value pairs OPT1, VAL1...
or in a struct OPTIONS with field names as option keys and field values
as option values:
FORMAT: data output format.
String setting the format of the output DATA. Valid values are:
'array': DATA is a matrix with sensor readings in the column order
specified by the SENSORS metadata field.
'struct': DATA is a struct with sensor names as field names
and column vectors of sensor readings as field values.
Default value: 'array'
TIMENAV: navigation data timestamp.
String setting the name of the time sensor for merging and sorting
sensor cycles from the navigation data set.
Default value: 'm_present_time'
TIMESCI: scientific data timestamp.
String setting the name of the time sensor for merging and sorting
sensor cycles from the science data set.
Default value: 'sci_m_present_time'
SENSORS: sensor filtering list.
String cell array with the names of the sensors of interest.
If given, only sensors present in both the input data sets and this
list will be present in output. The string 'all' may also be given,
in which case sensor filtering is not performed and all sensors
in the input data sets will be present in output.
Default value: 'all' (do not perform sensor filtering).
PERIOD: time filtering boundaries.
Two element numeric array with the start and the end of the period
of interest (seconds since 1970-01-01 00:0:00.00 UTC). If given,
only sensor cycles with timestamps within this period will be present
in output. The string 'all' may also be given, in which case time
filtering is not performed and all sensors cycles in the input
data sets will be present in output.
Default value: 'all' (do not perform time filtering).
Notes:
This function should be used to merge data from navigation and science
data sets, not from data sets coming from the same bay (use DBACAT
instead).
The function is designed after the programs provided by WRC 'dba_merge',
'dba_sensor_filter' and 'dba_time_filter'. Since these programs are not
well documented, specially 'dba_merge', and the source code is not
available, it is coded after some reverse engineering and its behaviour
only asserted for compatibility with original 'dba_merge' program through
experimental ways.
The merging process sorts sensor cycles from navigation and science data
sets comparing the respective timestamp values. Sensor cycles coming from
navigation and science data arrays with equal timestamp values are merged
into a single sensor cycle, otherwise the missing sensor values are filled
with invalid values (NaN). In addition, if some sensor is present in both
data sets, the repeated instance is renamed: 'sci_XXX' to 'gld_dup_sci_XXX'
and 'YYY' to 'sci_dup_YYY'. If the sensor originates in the science bay,
the navigation bay instance is prepended with the prefix 'gld_dup_'.
Otherwise if the sensor originates in the navigation bay, the science bay
instance is prepended with the prefix 'sci_dup_'. This renaming is the
behaviour of the original program 'dba_merge', as described here:
<http://marine.rutgers.edu/~kerfoot/slocum/data/readme/wrc_doco/dbd_file_format.txt>
Examples:
[meta, data] = dbamerge(meta_nav, data_nav, meta_sci, data_sci)
See also:
XBD2DBA
DBA2MAT
DBACAT
Authors:
Joan Pau Beltran <joanpau.beltran@socib.cat>0001 function [meta, data] = dbamerge(meta_nav, data_nav, meta_sci, data_sci, varargin) 0002 %DBAMERGE Merge data from combined navigation and science data sets into a single data set. 0003 % 0004 % Syntax: 0005 % [META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI) 0006 % [META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPTIONS) 0007 % [META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPT1, VAL1, ...) 0008 % 0009 % Description: 0010 % [META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI) merges the 0011 % navigation and science data sets described by metadata structs META_NAV and 0012 % META_SCI, and data arrays DATA_NAV and DATA_SCI into a single data set 0013 % described by metadata struct META and data array or struct DATA 0014 % (see format option described below). Input metadata and data should be 0015 % in the format returned by the function DBACAT. Sensor cycles from both 0016 % data sets are merged based on the order of the respective timestamps. 0017 % See note on merging process. 0018 % 0019 % [META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPTIONS) and 0020 % [META, DATA] = DBAMERGE(META_NAV, DATA_NAV, META_SCI, DATA_SCI, OPT1, VAL1, ...) 0021 % accept the following options given in key-value pairs OPT1, VAL1... 0022 % or in a struct OPTIONS with field names as option keys and field values 0023 % as option values: 0024 % FORMAT: data output format. 0025 % String setting the format of the output DATA. Valid values are: 0026 % 'array': DATA is a matrix with sensor readings in the column order 0027 % specified by the SENSORS metadata field. 0028 % 'struct': DATA is a struct with sensor names as field names 0029 % and column vectors of sensor readings as field values. 0030 % Default value: 'array' 0031 % TIMENAV: navigation data timestamp. 0032 % String setting the name of the time sensor for merging and sorting 0033 % sensor cycles from the navigation data set. 0034 % Default value: 'm_present_time' 0035 % TIMESCI: scientific data timestamp. 0036 % String setting the name of the time sensor for merging and sorting 0037 % sensor cycles from the science data set. 0038 % Default value: 'sci_m_present_time' 0039 % SENSORS: sensor filtering list. 0040 % String cell array with the names of the sensors of interest. 0041 % If given, only sensors present in both the input data sets and this 0042 % list will be present in output. The string 'all' may also be given, 0043 % in which case sensor filtering is not performed and all sensors 0044 % in the input data sets will be present in output. 0045 % Default value: 'all' (do not perform sensor filtering). 0046 % PERIOD: time filtering boundaries. 0047 % Two element numeric array with the start and the end of the period 0048 % of interest (seconds since 1970-01-01 00:0:00.00 UTC). If given, 0049 % only sensor cycles with timestamps within this period will be present 0050 % in output. The string 'all' may also be given, in which case time 0051 % filtering is not performed and all sensors cycles in the input 0052 % data sets will be present in output. 0053 % Default value: 'all' (do not perform time filtering). 0054 % 0055 % Notes: 0056 % This function should be used to merge data from navigation and science 0057 % data sets, not from data sets coming from the same bay (use DBACAT 0058 % instead). 0059 % 0060 % The function is designed after the programs provided by WRC 'dba_merge', 0061 % 'dba_sensor_filter' and 'dba_time_filter'. Since these programs are not 0062 % well documented, specially 'dba_merge', and the source code is not 0063 % available, it is coded after some reverse engineering and its behaviour 0064 % only asserted for compatibility with original 'dba_merge' program through 0065 % experimental ways. 0066 % 0067 % The merging process sorts sensor cycles from navigation and science data 0068 % sets comparing the respective timestamp values. Sensor cycles coming from 0069 % navigation and science data arrays with equal timestamp values are merged 0070 % into a single sensor cycle, otherwise the missing sensor values are filled 0071 % with invalid values (NaN). In addition, if some sensor is present in both 0072 % data sets, the repeated instance is renamed: 'sci_XXX' to 'gld_dup_sci_XXX' 0073 % and 'YYY' to 'sci_dup_YYY'. If the sensor originates in the science bay, 0074 % the navigation bay instance is prepended with the prefix 'gld_dup_'. 0075 % Otherwise if the sensor originates in the navigation bay, the science bay 0076 % instance is prepended with the prefix 'sci_dup_'. This renaming is the 0077 % behaviour of the original program 'dba_merge', as described here: 0078 % <http://marine.rutgers.edu/~kerfoot/slocum/data/readme/wrc_doco/dbd_file_format.txt> 0079 % 0080 % Examples: 0081 % [meta, data] = dbamerge(meta_nav, data_nav, meta_sci, data_sci) 0082 % 0083 % See also: 0084 % XBD2DBA 0085 % DBA2MAT 0086 % DBACAT 0087 % 0088 % Authors: 0089 % Joan Pau Beltran <joanpau.beltran@socib.cat> 0090 0091 % Copyright (C) 2013-2016 0092 % ICTS SOCIB - Servei d'observacio i prediccio costaner de les Illes Balears 0093 % <http://www.socib.es> 0094 % 0095 % This program is free software: you can redistribute it and/or modify 0096 % it under the terms of the GNU General Public License as published by 0097 % the Free Software Foundation, either version 3 of the License, or 0098 % (at your option) any later version. 0099 % 0100 % This program is distributed in the hope that it will be useful, 0101 % but WITHOUT ANY WARRANTY; without even the implied warranty of 0102 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 0103 % GNU General Public License for more details. 0104 % 0105 % You should have received a copy of the GNU General Public License 0106 % along with this program. If not, see <http://www.gnu.org/licenses/>. 0107 0108 error(nargchk(4, 14, nargin, 'struct')); 0109 0110 0111 %% Set options and default values. 0112 options.format = 'array'; 0113 options.timenav = 'm_present_time'; 0114 options.timesci = 'sci_m_present_time'; 0115 options.sensors = 'all'; 0116 options.period = 'all'; 0117 0118 0119 %% Parse optional arguments. 0120 % Get option key-value pairs in any accepted call signature. 0121 argopts = varargin; 0122 if isscalar(argopts) && isstruct(argopts{1}) 0123 % Options passed as a single option struct argument: 0124 % field names are option keys and field values are option values. 0125 opt_key_list = fieldnames(argopts{1}); 0126 opt_val_list = struct2cell(argopts{1}); 0127 elseif mod(numel(argopts), 2) == 0 0128 % Options passed as key-value argument pairs. 0129 opt_key_list = argopts(1:2:end); 0130 opt_val_list = argopts(2:2:end); 0131 else 0132 error('glider_toolbox:dbamerge:InvalidOptions', ... 0133 'Invalid optional arguments (neither key-value pairs nor struct).'); 0134 end 0135 % Overwrite default options with values given in extra arguments. 0136 for opt_idx = 1:numel(opt_key_list) 0137 opt = lower(opt_key_list{opt_idx}); 0138 val = opt_val_list{opt_idx}; 0139 if isfield(options, opt) 0140 options.(opt) = val; 0141 else 0142 error('glider_toolbox:dbamerge:InvalidOption', ... 0143 'Invalid option: %s.', opt); 0144 end 0145 end 0146 0147 0148 %% Set option flags and values. 0149 output_format = lower(options.format); 0150 time_sensor_nav = options.timenav; 0151 time_sensor_sci = options.timesci; 0152 sensor_filtering = true; 0153 sensor_list = cellstr(options.sensors); 0154 time_filtering = true; 0155 time_range = options.period; 0156 if ischar(options.sensors) && strcmp(options.sensors, 'all') 0157 sensor_filtering = false; 0158 end 0159 if ischar(options.period) && strcmp(options.period, 'all') 0160 time_filtering = false; 0161 end 0162 0163 0164 %% Merge data and metadata checking for empty input cases. 0165 if isempty(meta_sci.sources) && isempty(meta_nav.sources) 0166 % No input data. 0167 % Both META_NAV and DATA_NAV, and META_SCI and DATA_SCI 0168 % are equal to the trivial output of DBACAT. 0169 % Disable filtering. 0170 meta = meta_nav; 0171 data = data_nav; 0172 sensor_filtering = false; 0173 time_filtering = false; 0174 elseif isempty(meta_sci.sources) 0175 % Only navigation data. 0176 meta = meta_nav; 0177 data = data_nav; 0178 time_sensor_merged = time_sensor_nav; % Time sensor for filtering. 0179 elseif isempty(meta_nav.sources) 0180 % Only science data. 0181 meta = meta_sci; 0182 data = data_sci; 0183 time_sensor_merged = time_sensor_sci; % Time sensor for filtering. 0184 else 0185 % Merge metadata performing sensor renaming if needed. 0186 % Sensor renaming is done to mimic the behaviour of WRC program 'dba_merge'. 0187 sources_nav = meta_nav.sources; 0188 headers_nav = meta_nav.headers; 0189 sensors_nav = meta_nav.sensors; 0190 units_nav = meta_nav.units; 0191 bytes_nav = meta_nav.bytes; 0192 sources_sci = meta_sci.sources; 0193 headers_sci = meta_sci.headers; 0194 sensors_sci = meta_sci.sensors; 0195 units_sci = meta_sci.units; 0196 bytes_sci = meta_sci.bytes; 0197 [sensors_dup, sensors_dup_index_nav, sensors_dup_index_sci] = ... 0198 intersect(sensors_nav, sensors_sci); 0199 sensors_dup_sci_select = strncmp('sci_', sensors_dup, 4); 0200 sensors_dup_nav_select = ~sensors_dup_sci_select; 0201 sensors_nav(sensors_dup_index_nav(sensors_dup_sci_select)) = ... 0202 strcat('gld_dup_', sensors_dup(sensors_dup_sci_select)); 0203 sensors_sci(sensors_dup_index_sci(sensors_dup_nav_select)) = ... 0204 strcat('sci_dup_', sensors_dup(sensors_dup_nav_select)); 0205 meta.sources = vertcat(sources_nav, sources_sci); 0206 meta.headers = vertcat(headers_nav, headers_sci); 0207 meta.sensors = vertcat(sensors_nav, sensors_sci); 0208 meta.units = vertcat(units_nav, units_sci); 0209 meta.bytes = vertcat(bytes_nav, bytes_sci); 0210 0211 % Merge data. 0212 % Check that both data sets have their own timestamp sensor. 0213 [time_sensor_nav_present, time_sensor_nav_col] = ... 0214 ismember(time_sensor_nav, sensors_nav); 0215 if ~time_sensor_nav_present 0216 error('glider_toolbox:dbamerge:MissingTimestamp', ... 0217 'Missing timestamp sensor in navigation data set: %s.', ... 0218 time_sensor_nav); 0219 end 0220 [time_sensor_sci_present, time_sensor_sci_col] = ... 0221 ismember(time_sensor_sci, sensors_sci); 0222 if ~time_sensor_sci_present 0223 error('glider_toolbox:dbamerge:MissingTimestamp', ... 0224 'Missing timestamp sensor in science data set: %s.', ... 0225 time_sensor_sci); 0226 end 0227 0228 % Build list of unique timestamps and output index of each sensor cycle. 0229 stamp_nav = data_nav(:, time_sensor_nav_col); 0230 stamp_sci = data_sci(:, time_sensor_sci_col); 0231 [stamp_merged, ~, stamp_merged_indices_to] = ... 0232 unique(vertcat(stamp_nav, stamp_sci)); 0233 0234 % Build merged data array with sensor columns horizontally concatenated 0235 % and different sensor cycles verticaly interleaved according to timestamp. 0236 row_num_nav = numel(stamp_nav); 0237 row_range_nav = (1:row_num_nav); 0238 row_num_sci = numel(stamp_sci); 0239 row_range_sci = row_num_nav + (1:row_num_sci); 0240 row_num_merged = numel(stamp_merged); 0241 col_num_nav = numel(sensors_nav); 0242 col_range_nav = (1:col_num_nav); 0243 col_num_sci = numel(sensors_sci); 0244 col_range_sci = col_num_nav + (1:col_num_sci); 0245 col_num_merged = col_num_nav + col_num_sci; 0246 data = nan(row_num_merged, col_num_merged); 0247 data(stamp_merged_indices_to(row_range_nav), col_range_nav) = data_nav; 0248 data(stamp_merged_indices_to(row_range_sci), col_range_sci) = data_sci; 0249 0250 % Fill missing navigation timestamp values with science timestamp values. 0251 % This is done to mimic the behaviour of the WRC program 'dba_merge'. 0252 stamp_nav_invalid = isnan(data(:, time_sensor_nav_col)); 0253 data(stamp_nav_invalid, time_sensor_nav_col) = ... 0254 data(stamp_nav_invalid, col_num_nav + time_sensor_sci_col); 0255 0256 % Unique timestamp to be used for time filtering. 0257 time_sensor_merged = time_sensor_nav; 0258 end 0259 0260 0261 %% Perform time filtering if needed. 0262 if time_filtering 0263 [time_sensor_merged_present, time_sensor_merged_col] = ... 0264 ismember(time_sensor_merged, meta.sensors); 0265 if ~time_sensor_merged_present 0266 error('glider_toolbox:dbamerge:MissingTimestamp', ... 0267 'Missing timestamp sensor in merged data set: %s.', ... 0268 time_sensor_merged); 0269 end 0270 stamp_merged = data(:, time_sensor_merged_col); 0271 stamp_select = ... 0272 ~(stamp_merged < time_range(1) | stamp_merged > time_range(2)); 0273 data = data(stamp_select, :); 0274 end 0275 0276 0277 %% Perform sensor filtering if needed. 0278 if sensor_filtering 0279 [sensor_select, ~] = ismember(meta.sensors, sensor_list); 0280 meta.sensors = meta.sensors(sensor_select); 0281 meta.units = meta.units(sensor_select); 0282 meta.bytes = meta.bytes(sensor_select); 0283 data = data(:, sensor_select); 0284 end 0285 0286 0287 %% Convert output data to struct format if needed. 0288 switch output_format 0289 case 'array' 0290 case 'struct' 0291 data = cell2struct(num2cell(data, 1), meta.sensors, 2); 0292 otherwise 0293 error('glider_toolbox:dbamerge:InvalidFormat', ... 0294 'Invalid output format: %s.', output_format) 0295 end 0296 0297 end