LOADSEAGLIDERDATA Load Seaglider data from log and eng files in directory.
Syntax:
[META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP)
[META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPTIONS)
[META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPT1, VAL1, ...)
Description:
[META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP) loads data
and metadata from Seaglider files in log text format (.log) and eng data
format (.eng) contained in directory named by string SGDIR and whose name
matches regular expression in string LOGREGEXP or string ENGREGEXP.
META and DATA contain loaded metadata and data in the format returned by
SGLOGENGMERGE.
[META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPTIONS) and
[META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPT1, VAL1, ...)
accept the following options, given in key-value pairs OPT1, VAL1...
or in a struct OPTIONS with field names as option keys and field values
as option values, allowing to restrict the time range or the set of
data parameters and columns to load, or to specify the output format:
FORMAT: data output format.
String setting the format of the output DATA. Valid values are:
'array': DATA is a struct with the following fields:
LOG: array with log parameters in the column order specified by
PARAMS metadata field.
ENG: array or cell array with the eng columns and GC and SM_CCo
log parameter columns in the column order specified by COLUMNS
metadata field.
GPS: array or cell array with the GPS log columns in the order
specified by the GPS metadata field.
See note on array format.
'merged': DATA is a struct with a column vector field for each
parameter in log or gps data, and for each column in eng data.
'struct': DATA is a struct with a column vector field for each
column in eng data and gps data, and for each scalar parameter
in log data; and with a struct field for each non-scalar parameter
in log data.
Default value: 'array'
PARAMS: log parameter filtering list.
String cell array with the names of the log parameters of interest.
If given, only parameters present in both the input data set and
this list will be present in output. For non-scalar parameters,
the name of the identifier as it appears in the log line specifies
including all of its fields. Individual parameter fields are selected
with the identifier and the name of the field separated by underscore
(e.g. 'FINISH_dens'). The string 'all' may also be given,
in which case parameter filtering is not performed and all parameters
in input list will be present in output.
Default value: 'all' (do not perform log parameter filtering).
COLUMNS: eng column filtering list.
String cell array with the names of the eng data columns of interest.
If given, only columns present in both the input data set and this list
will be present in output. The string 'all' may also be given,
in which case column filtering is not performed and all columns
in input list will be present in output.
Default value: 'all' (do not perform eng column filtering).
PERIOD: dive start time filtering boundaries.
Two element numeric array with the start and the end of the period
of interest (seconds since 1970-01-01 00:00:00.00 UTC). If given,
only data from dives with start time within this period will be
present in output. The string 'all' may also be given, in which case
time filtering is not performed and data from all dives will be
present in output.
Default value: 'all' (do not perform time filtering).
Notes:
This function is a simple shortcut to load all log and eng data in a
directory belonging to the same deployment or transect. It just filters
the contents of the directory and calls SGLOG2MAT and SGLOGCAT, SGENG2MAT
and SGENGCAT, and finally SGLOGENGMERGE, bypassing the given options.
When there are both textual and numeric columns in the output,
the 'array' format might require a large amount of memory because of the
storage in a cell array. Formats 'struct' and 'merged' require much
less space in that case.
Examples:
[meta, data] = ...
loadSeagliderData(sgdir, logregexp, engregexp)
[meta, data] = ...
loadSeagliderData(sgdir, logregexp, engregexp, ...
'columns', columns_of_interest, ...
'params', params_of_interest,
'period', period_of_interest, ...
'format', 'merged');
See also:
SGLOG2MAT
SGLOGCAT
SGENG2MAT
SGENGCAT
SGLOGENGMERGE
DIR
REGEXP
Authors:
Joan Pau Beltran <joanpau.beltran@socib.cat>0001 function [meta, data] = loadSeagliderData(sgdir, logregexp, engregexp, varargin) 0002 %LOADSEAGLIDERDATA Load Seaglider data from log and eng files in directory. 0003 % 0004 % Syntax: 0005 % [META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP) 0006 % [META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPTIONS) 0007 % [META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPT1, VAL1, ...) 0008 % 0009 % Description: 0010 % [META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP) loads data 0011 % and metadata from Seaglider files in log text format (.log) and eng data 0012 % format (.eng) contained in directory named by string SGDIR and whose name 0013 % matches regular expression in string LOGREGEXP or string ENGREGEXP. 0014 % META and DATA contain loaded metadata and data in the format returned by 0015 % SGLOGENGMERGE. 0016 % 0017 % [META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPTIONS) and 0018 % [META, DATA] = LOADSEAGLIDERDATA(SGDIR, LOGREGEXP, ENGREGEXP, OPT1, VAL1, ...) 0019 % accept the following options, given in key-value pairs OPT1, VAL1... 0020 % or in a struct OPTIONS with field names as option keys and field values 0021 % as option values, allowing to restrict the time range or the set of 0022 % data parameters and columns to load, or to specify the output format: 0023 % FORMAT: data output format. 0024 % String setting the format of the output DATA. Valid values are: 0025 % 'array': DATA is a struct with the following fields: 0026 % LOG: array with log parameters in the column order specified by 0027 % PARAMS metadata field. 0028 % ENG: array or cell array with the eng columns and GC and SM_CCo 0029 % log parameter columns in the column order specified by COLUMNS 0030 % metadata field. 0031 % GPS: array or cell array with the GPS log columns in the order 0032 % specified by the GPS metadata field. 0033 % See note on array format. 0034 % 'merged': DATA is a struct with a column vector field for each 0035 % parameter in log or gps data, and for each column in eng data. 0036 % 'struct': DATA is a struct with a column vector field for each 0037 % column in eng data and gps data, and for each scalar parameter 0038 % in log data; and with a struct field for each non-scalar parameter 0039 % in log data. 0040 % Default value: 'array' 0041 % PARAMS: log parameter filtering list. 0042 % String cell array with the names of the log parameters of interest. 0043 % If given, only parameters present in both the input data set and 0044 % this list will be present in output. For non-scalar parameters, 0045 % the name of the identifier as it appears in the log line specifies 0046 % including all of its fields. Individual parameter fields are selected 0047 % with the identifier and the name of the field separated by underscore 0048 % (e.g. 'FINISH_dens'). The string 'all' may also be given, 0049 % in which case parameter filtering is not performed and all parameters 0050 % in input list will be present in output. 0051 % Default value: 'all' (do not perform log parameter filtering). 0052 % COLUMNS: eng column filtering list. 0053 % String cell array with the names of the eng data columns of interest. 0054 % If given, only columns present in both the input data set and this list 0055 % will be present in output. The string 'all' may also be given, 0056 % in which case column filtering is not performed and all columns 0057 % in input list will be present in output. 0058 % Default value: 'all' (do not perform eng column filtering). 0059 % PERIOD: dive start time filtering boundaries. 0060 % Two element numeric array with the start and the end of the period 0061 % of interest (seconds since 1970-01-01 00:00:00.00 UTC). If given, 0062 % only data from dives with start time within this period will be 0063 % present in output. The string 'all' may also be given, in which case 0064 % time filtering is not performed and data from all dives will be 0065 % present in output. 0066 % Default value: 'all' (do not perform time filtering). 0067 % 0068 % Notes: 0069 % This function is a simple shortcut to load all log and eng data in a 0070 % directory belonging to the same deployment or transect. It just filters 0071 % the contents of the directory and calls SGLOG2MAT and SGLOGCAT, SGENG2MAT 0072 % and SGENGCAT, and finally SGLOGENGMERGE, bypassing the given options. 0073 % 0074 % When there are both textual and numeric columns in the output, 0075 % the 'array' format might require a large amount of memory because of the 0076 % storage in a cell array. Formats 'struct' and 'merged' require much 0077 % less space in that case. 0078 % 0079 % Examples: 0080 % [meta, data] = ... 0081 % loadSeagliderData(sgdir, logregexp, engregexp) 0082 % [meta, data] = ... 0083 % loadSeagliderData(sgdir, logregexp, engregexp, ... 0084 % 'columns', columns_of_interest, ... 0085 % 'params', params_of_interest, 0086 % 'period', period_of_interest, ... 0087 % 'format', 'merged'); 0088 % 0089 % See also: 0090 % SGLOG2MAT 0091 % SGLOGCAT 0092 % SGENG2MAT 0093 % SGENGCAT 0094 % SGLOGENGMERGE 0095 % DIR 0096 % REGEXP 0097 % 0098 % Authors: 0099 % Joan Pau Beltran <joanpau.beltran@socib.cat> 0100 0101 % Copyright (C) 2013-2016 0102 % ICTS SOCIB - Servei d'observacio i prediccio costaner de les Illes Balears 0103 % <http://www.socib.es> 0104 % 0105 % This program is free software: you can redistribute it and/or modify 0106 % it under the terms of the GNU General Public License as published by 0107 % the Free Software Foundation, either version 3 of the License, or 0108 % (at your option) any later version. 0109 % 0110 % This program is distributed in the hope that it will be useful, 0111 % but WITHOUT ANY WARRANTY; without even the implied warranty of 0112 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 0113 % GNU General Public License for more details. 0114 % 0115 % You should have received a copy of the GNU General Public License 0116 % along with this program. If not, see <http://www.gnu.org/licenses/>. 0117 0118 error(nargchk(3, 13, nargin, 'struct')); 0119 0120 0121 %% Set options and default values. 0122 options.format = 'array'; 0123 options.columns = 'all'; 0124 options.params = 'all'; 0125 options.period = 'all'; 0126 0127 0128 %% Parse optional arguments. 0129 % Get option key-value pairs in any accepted call signature. 0130 argopts = varargin; 0131 if isscalar(argopts) && isstruct(argopts{1}) 0132 % Options passed as a single option struct argument: 0133 % field names are option keys and field values are option values. 0134 opt_key_list = fieldnames(argopts{1}); 0135 opt_val_list = struct2cell(argopts{1}); 0136 elseif mod(numel(argopts), 2) == 0 0137 % Options passed as key-value argument pairs. 0138 opt_key_list = argopts(1:2:end); 0139 opt_val_list = argopts(2:2:end); 0140 else 0141 error('glider_toolbox:loadSeagliderData:InvalidOptions', ... 0142 'Invalid optional arguments (neither key-value pairs nor struct).'); 0143 end 0144 % Overwrite default options with values given in extra arguments. 0145 for opt_idx = 1:numel(opt_key_list) 0146 opt = lower(opt_key_list{opt_idx}); 0147 val = opt_val_list{opt_idx}; 0148 if isfield(options, opt) 0149 options.(opt) = val; 0150 else 0151 error('glider_toolbox:loadSeagliderData:InvalidOption', ... 0152 'Invalid option: %s.', opt); 0153 end 0154 end 0155 0156 0157 %% Get file names matching the desired patterns. 0158 % Flatten lists to discard unmatched files. 0159 sgdir_contents = dir(sgdir); 0160 log_sel = ~[sgdir_contents.isdir] ... 0161 & ~cellfun(@isempty, regexp({sgdir_contents.name}, logregexp)); 0162 eng_sel = ~[sgdir_contents.isdir] ... 0163 & ~cellfun(@isempty, regexp({sgdir_contents.name}, engregexp)); 0164 log_names = {sgdir_contents(log_sel).name}; 0165 log_sizes = [sgdir_contents(log_sel).bytes]; 0166 eng_names = {sgdir_contents(eng_sel).name}; 0167 eng_sizes = [sgdir_contents(eng_sel).bytes]; 0168 disp(['Seaglider log files found: ' num2str(numel(log_names)) ... 0169 ' (' num2str(sum(log_sizes)*2^-10) ' kB).']); 0170 disp(['Seaglider eng files found: ' num2str(numel(eng_names)) ... 0171 ' (' num2str(sum(eng_sizes)*2^-10) ' kB).']); 0172 0173 0174 %% Load log files. 0175 disp('Loading Seaglider log files...'); 0176 log_files = cell(size(log_names)); 0177 log_success = false(size(log_names)); 0178 meta_log = cell(size(log_names)); 0179 data_log = cell(size(log_names)); 0180 for log_idx = 1:numel(log_names) 0181 try 0182 log_files{log_idx} = fullfile(sgdir, log_names{log_idx}); 0183 [meta_log{log_idx}, data_log{log_idx}] = ... 0184 sglog2mat(log_files{log_idx}, 'params', options.params); 0185 log_success(log_idx) = true; 0186 catch exception 0187 disp(['Error loading Seaglider log file ' log_files{log_idx} ':']); 0188 disp(getReport(exception, 'extended')); 0189 end 0190 end 0191 meta_log = meta_log(log_success); 0192 data_log = data_log(log_success); 0193 disp(['Seaglider log files loaded: ' ... 0194 num2str(numel(data_log)) ' of ' num2str(numel(log_names)) '.']); 0195 0196 0197 %% Load eng files. 0198 disp('Loading Seaglider eng files...'); 0199 eng_files = cell(size(eng_names)); 0200 eng_success = false(size(eng_names)); 0201 meta_eng = cell(size(eng_names)); 0202 data_eng = cell(size(eng_names)); 0203 for eng_idx = 1:numel(eng_names) 0204 try 0205 eng_files{eng_idx} = fullfile(sgdir, eng_names{eng_idx}); 0206 [meta_eng{eng_idx}, data_eng{eng_idx}] = ... 0207 sgeng2mat(eng_files{eng_idx}, 'columns', options.columns); 0208 eng_success(eng_idx) = true; 0209 catch exception 0210 disp(['Error loading Seglider eng file ' eng_files{eng_idx} ':']); 0211 disp(getReport(exception, 'extended')); 0212 end 0213 end 0214 meta_eng = meta_eng(eng_success); 0215 data_eng = data_eng(eng_success); 0216 disp(['Seaglider eng files loaded: ' ... 0217 num2str(numel(data_eng)) ' of ' num2str(numel(eng_names)) '.']); 0218 0219 0220 %% Combine data from log and eng files respectively. 0221 [meta_log, data_log] = sglogcat(meta_log, data_log, 'period', options.period); 0222 [meta_eng, data_eng] = sgengcat(meta_eng, data_eng, 'period', options.period); 0223 0224 0225 %% Merge data from log and eng files. 0226 [meta, data] = sglogengmerge(meta_log, data_log, meta_eng, data_eng, ... 0227 'format', options.format); 0228 0229 end