MAIN_GLIDER_DATA_PROCESSING_DT Run delayed time glider processing chain.
Description:
This script develops the processing chain for delayed time glider data:
- Check for configured deployments to process in delayed mode.
- Convert deployment binary files to human readable format, if needed.
- Load data from all files in a single and consistent structure.
- Generate standarized product version of raw data (NetCDF level 0).
- Preprocess raw data applying simple unit conversions and factory
calibrations without modifying their nominal value:
-- Select reference sensors for time and space coordinates.
Perform unit conversions if necessary.
-- Select extra navigation sensors: waypoints, pitch, depth...
Perform unit conversions if necessary.
-- Select sensors of interest: CTD, oxygen, ocean color...
Perform unit conversions and factory calibrations if necessary.
- Process preprocessed data to obtain well referenced trajectory data
with new derived measurements and corrections:
-- Fill missing values of time and space reference sensors.
-- Fill missing values of other navigation sensors.
-- Identify transect boundaries at waypoint changes.
-- Identify cast boundaries from vertical direction changes.
-- Apply generic sensor processings: sensor lag correction...
-- Process CTD data: pressure filtering, thermal lag correction...
-- Derive new measurements: depth, salinity, density...
- Generate standarized product version of trajectory data (NetCDF
level 1).
- Generate descriptive figures from trajectory data.
- Interpolate/bin trajectory data to obtain gridded data (vertical
instantaneous profiles of already processed data).
- Generate standarized product version of gridded data (NetCDF level 2).
- Generate descriptive figures from gridded data.
- Copy generated data products to its public location, if needed.
- Copy generated figures to its public location and generate figure
information service file, if needed.
Deployment information is queried from a data base by GETDEPLOYMENTINFODB.
Data base access parameters may be configured in CONFIGDBACCESS.
Selected deployments and their metadata fields may be configured in
CONFIGDTDEPLOYMENTINFOQUERYDB.
For each deployment, the messages produced during each processing step are
recorded to a log file. This recording is enabled just before the start of
the processing of the deployment, and it is turned off when the processing
finishes, with the function DIARY.
Input deployment raw data is loaded from a directory of raw text files
with LOADSLOCUMDATA, LOADSEAGLIDERDATA or LOADSEAEXPLORERDATA.
For Slocum gliders a directory of raw binary files may also
be specified, and automatic conversion to text file format may be enabled.
The conversion is performed by the function XBD2DBA, which is called
for each binary file in the specified binary directory
with a renaming pattern to specify the name of the resulting text file,
and performs a system call to program 'dbd2asc' by WRC.
The path to the 'dbd2asc' program may be configured in CONFIGWRCPROGRAMS.
Input file conversion and data loading options may be configured in
CONFIGDTFILEOPTIONSSLOCUM, CONFIGDTFILEOPTIONSSEAGLIDER, and
CONFIGDTFILEOPTIONSSEAEXPLORER.
Output products, figures and processing logs are generated to local paths.
Input and output paths may be configured using expressions built upon
deployment field value replacements in CONFIGDTPATHSLOCAL.
Raw data is preprocessed to apply some simple unit conversions with the
function PREPROCESSGLIDERDATA. The preprocessing options and its
parameters may be configured in CONFIGDATAPREPROCESSINGSLOCUM,
CONFIGDATAPREPROCESSINGSEAGLIDER and CONFIGDATAPREPROCESSINGSEAEXPLORER.
Preprocessed data is processed with PROCESSGLIDERDATA to obtain properly
referenced data with a trajectory data structure. The desired processing
actions (interpolations, filterings, corrections and derivations)
and its parameters may be configured in CONFIGDATAPROCESSINGSLOCUMG1,
CONFIGDATAPROCESSINGSLOCUMG2, CONFIGDATAPROCESSINGSEAGLIDER and
CONFIGDATAPROCESSINGSEAEXPLORER.
Processed data is interpolated/binned with GRIDGLIDERDATA to obtain a data
set with the structure of a trajectory of instantaneous vertical profiles
sampled at a common set of regular depth levels. The desired gridding
parameters may be configured in CONFIGDATAGRIDDING.
Standard products in NetCDF format are generated from raw data,
processed data and gridded data with GENERATEOUTPUTNETCDF.
Raw data is stored in NetCDF format as level 0 output product.
This file mimics the appearance of the raw data text files, but gathering
all useful data in a single place. Hence, the structure of the resulting
NetCDF file varies with each type of glider, and may be configured
in CONFIGDTOUTPUTNETCDFL0SLOCUM, CONFIGDTOUTPUTNETCDFL0SEAGLIDER and
CONFIGDTOUTPUTNETCDFL0SEAEXPLORER. Processed and gridded data sets are
stored in NetCDF format as level 1 and level 2 output products
respectively. The structure of these files does not depend on the type
of glider the data comes from, and may be configured in
CONFIGDTOUTPUNETCDFL1 and CONFIGDTOUTPUTNETCDFL2 respectively.
Figures describing the collected glider data may be generated from
processed data and from gridded data. Figures are generated by
GENERATEGLIDERFIGURES, and may be configured in CONFIGFIGURES.
Available plots are: scatter plots of measurements on vertical transect
sections, temperature-salinity diagrams, trajectory and current maps,
and profile statistics plots. Other plot functions may be used,
provided that their call syntax is compatible with GENERATEGLIDERFIGURES.
Selected data output products and figures may be copied to a public
location for distribution purposes. For figures, a service file describing
the available figures and their public location may also be generated.
This file is generated by function SAVEJSON with the figure information
returned by GENERATEGLIDERFIGURES updated with the new public location.
Public products and figures to copy and their locations may be configured
in CONFIGDTPATHSPUBLIC.
See also:
CONFIGWRCPROGRAMS
CONFIGDBACCESS
CONFIGDTDEPLOYMENTINFOQUERYDB
CONFIGDTPATHSLOCAL
CONFIGDTFILEOPTIONSSLOCUM
CONFIGDTFILEOPTIONSSEAGLIDER
CONFIGDTFILEOPTIONSSEAEXPLORER
CONFIGDATAPREPROCESSINGSLOCUM
CONFIGDATAPREPROCESSINGSEAGLIDER
CONFIGDATAPREPROCESSINGSEAEXPLORER
CONFIGDATAPROCESSINGSLOCUMG1
CONFIGDATAPROCESSINGSLOCUMG2
CONFIGDATAPROCESSINGSEAGLIDER
CONFIGDATAPROCESSINGSEAEXPLORER
CONFIGDATAGRIDDING
CONFIGDTOUTPUTNETCDFL0SLOCUM
CONFIGDTOUTPUTNETCDFL0SEAGLIDER
CONFIGDTOUTPUTNETCDFL0SEAEXPLORER
CONFIGDTOUTPUTNETCDFL1
CONFIGDTOUTPUTNETCDFL2
CONFIGFIGURES
GETDEPLOYMENTINFODB
LOADSLOCUMDATA
PREPROCESSGLIDERDATA
PROCESSGLIDERDATA
GRIDGLIDERDATA
GENERATEOUTPUTNETCDF
GENERATEFIGURES
DIARY
STRFSTRUCT
XBD2DBA
SAVEJSON
Notes:
This script is based on the previous work by Tomeu Garau. He is the true
glider man.
Authors:
Frederic Cyr <Frederic.Cyr@mio.osupytheas.fr>
Joan Pau Beltran <joanpau.beltran@socib.cat>main_glider_data_processing_dt.m
0001 %MAIN_GLIDER_DATA_PROCESSING_DT Run delayed time glider processing chain. 0002 % 0003 % Description: 0004 % This script develops the processing chain for delayed time glider data: 0005 % - Check for configured deployments to process in delayed mode. 0006 % - Convert deployment binary files to human readable format, if needed. 0007 % - Load data from all files in a single and consistent structure. 0008 % - Generate standarized product version of raw data (NetCDF level 0). 0009 % - Preprocess raw data applying simple unit conversions and factory 0010 % calibrations without modifying their nominal value: 0011 % -- Select reference sensors for time and space coordinates. 0012 % Perform unit conversions if necessary. 0013 % -- Select extra navigation sensors: waypoints, pitch, depth... 0014 % Perform unit conversions if necessary. 0015 % -- Select sensors of interest: CTD, oxygen, ocean color... 0016 % Perform unit conversions and factory calibrations if necessary. 0017 % - Process preprocessed data to obtain well referenced trajectory data 0018 % with new derived measurements and corrections: 0019 % -- Fill missing values of time and space reference sensors. 0020 % -- Fill missing values of other navigation sensors. 0021 % -- Identify transect boundaries at waypoint changes. 0022 % -- Identify cast boundaries from vertical direction changes. 0023 % -- Apply generic sensor processings: sensor lag correction... 0024 % -- Process CTD data: pressure filtering, thermal lag correction... 0025 % -- Derive new measurements: depth, salinity, density... 0026 % - Generate standarized product version of trajectory data (NetCDF 0027 % level 1). 0028 % - Generate descriptive figures from trajectory data. 0029 % - Interpolate/bin trajectory data to obtain gridded data (vertical 0030 % instantaneous profiles of already processed data). 0031 % - Generate standarized product version of gridded data (NetCDF level 2). 0032 % - Generate descriptive figures from gridded data. 0033 % - Copy generated data products to its public location, if needed. 0034 % - Copy generated figures to its public location and generate figure 0035 % information service file, if needed. 0036 % 0037 % Deployment information is queried from a data base by GETDEPLOYMENTINFODB. 0038 % Data base access parameters may be configured in CONFIGDBACCESS. 0039 % Selected deployments and their metadata fields may be configured in 0040 % CONFIGDTDEPLOYMENTINFOQUERYDB. 0041 % 0042 % For each deployment, the messages produced during each processing step are 0043 % recorded to a log file. This recording is enabled just before the start of 0044 % the processing of the deployment, and it is turned off when the processing 0045 % finishes, with the function DIARY. 0046 % 0047 % Input deployment raw data is loaded from a directory of raw text files 0048 % with LOADSLOCUMDATA, LOADSEAGLIDERDATA or LOADSEAEXPLORERDATA. 0049 % For Slocum gliders a directory of raw binary files may also 0050 % be specified, and automatic conversion to text file format may be enabled. 0051 % The conversion is performed by the function XBD2DBA, which is called 0052 % for each binary file in the specified binary directory 0053 % with a renaming pattern to specify the name of the resulting text file, 0054 % and performs a system call to program 'dbd2asc' by WRC. 0055 % The path to the 'dbd2asc' program may be configured in CONFIGWRCPROGRAMS. 0056 % Input file conversion and data loading options may be configured in 0057 % CONFIGDTFILEOPTIONSSLOCUM, CONFIGDTFILEOPTIONSSEAGLIDER, and 0058 % CONFIGDTFILEOPTIONSSEAEXPLORER. 0059 % 0060 % Output products, figures and processing logs are generated to local paths. 0061 % Input and output paths may be configured using expressions built upon 0062 % deployment field value replacements in CONFIGDTPATHSLOCAL. 0063 % 0064 % Raw data is preprocessed to apply some simple unit conversions with the 0065 % function PREPROCESSGLIDERDATA. The preprocessing options and its 0066 % parameters may be configured in CONFIGDATAPREPROCESSINGSLOCUM, 0067 % CONFIGDATAPREPROCESSINGSEAGLIDER and CONFIGDATAPREPROCESSINGSEAEXPLORER. 0068 % 0069 % Preprocessed data is processed with PROCESSGLIDERDATA to obtain properly 0070 % referenced data with a trajectory data structure. The desired processing 0071 % actions (interpolations, filterings, corrections and derivations) 0072 % and its parameters may be configured in CONFIGDATAPROCESSINGSLOCUMG1, 0073 % CONFIGDATAPROCESSINGSLOCUMG2, CONFIGDATAPROCESSINGSEAGLIDER and 0074 % CONFIGDATAPROCESSINGSEAEXPLORER. 0075 % 0076 % Processed data is interpolated/binned with GRIDGLIDERDATA to obtain a data 0077 % set with the structure of a trajectory of instantaneous vertical profiles 0078 % sampled at a common set of regular depth levels. The desired gridding 0079 % parameters may be configured in CONFIGDATAGRIDDING. 0080 % 0081 % Standard products in NetCDF format are generated from raw data, 0082 % processed data and gridded data with GENERATEOUTPUTNETCDF. 0083 % Raw data is stored in NetCDF format as level 0 output product. 0084 % This file mimics the appearance of the raw data text files, but gathering 0085 % all useful data in a single place. Hence, the structure of the resulting 0086 % NetCDF file varies with each type of glider, and may be configured 0087 % in CONFIGDTOUTPUTNETCDFL0SLOCUM, CONFIGDTOUTPUTNETCDFL0SEAGLIDER and 0088 % CONFIGDTOUTPUTNETCDFL0SEAEXPLORER. Processed and gridded data sets are 0089 % stored in NetCDF format as level 1 and level 2 output products 0090 % respectively. The structure of these files does not depend on the type 0091 % of glider the data comes from, and may be configured in 0092 % CONFIGDTOUTPUNETCDFL1 and CONFIGDTOUTPUTNETCDFL2 respectively. 0093 % 0094 % Figures describing the collected glider data may be generated from 0095 % processed data and from gridded data. Figures are generated by 0096 % GENERATEGLIDERFIGURES, and may be configured in CONFIGFIGURES. 0097 % Available plots are: scatter plots of measurements on vertical transect 0098 % sections, temperature-salinity diagrams, trajectory and current maps, 0099 % and profile statistics plots. Other plot functions may be used, 0100 % provided that their call syntax is compatible with GENERATEGLIDERFIGURES. 0101 % 0102 % Selected data output products and figures may be copied to a public 0103 % location for distribution purposes. For figures, a service file describing 0104 % the available figures and their public location may also be generated. 0105 % This file is generated by function SAVEJSON with the figure information 0106 % returned by GENERATEGLIDERFIGURES updated with the new public location. 0107 % Public products and figures to copy and their locations may be configured 0108 % in CONFIGDTPATHSPUBLIC. 0109 % 0110 % See also: 0111 % CONFIGWRCPROGRAMS 0112 % CONFIGDBACCESS 0113 % CONFIGDTDEPLOYMENTINFOQUERYDB 0114 % CONFIGDTPATHSLOCAL 0115 % CONFIGDTFILEOPTIONSSLOCUM 0116 % CONFIGDTFILEOPTIONSSEAGLIDER 0117 % CONFIGDTFILEOPTIONSSEAEXPLORER 0118 % CONFIGDATAPREPROCESSINGSLOCUM 0119 % CONFIGDATAPREPROCESSINGSEAGLIDER 0120 % CONFIGDATAPREPROCESSINGSEAEXPLORER 0121 % CONFIGDATAPROCESSINGSLOCUMG1 0122 % CONFIGDATAPROCESSINGSLOCUMG2 0123 % CONFIGDATAPROCESSINGSEAGLIDER 0124 % CONFIGDATAPROCESSINGSEAEXPLORER 0125 % CONFIGDATAGRIDDING 0126 % CONFIGDTOUTPUTNETCDFL0SLOCUM 0127 % CONFIGDTOUTPUTNETCDFL0SEAGLIDER 0128 % CONFIGDTOUTPUTNETCDFL0SEAEXPLORER 0129 % CONFIGDTOUTPUTNETCDFL1 0130 % CONFIGDTOUTPUTNETCDFL2 0131 % CONFIGFIGURES 0132 % GETDEPLOYMENTINFODB 0133 % LOADSLOCUMDATA 0134 % PREPROCESSGLIDERDATA 0135 % PROCESSGLIDERDATA 0136 % GRIDGLIDERDATA 0137 % GENERATEOUTPUTNETCDF 0138 % GENERATEFIGURES 0139 % DIARY 0140 % STRFSTRUCT 0141 % XBD2DBA 0142 % SAVEJSON 0143 % 0144 % Notes: 0145 % This script is based on the previous work by Tomeu Garau. He is the true 0146 % glider man. 0147 % 0148 % Authors: 0149 % Frederic Cyr <Frederic.Cyr@mio.osupytheas.fr> 0150 % Joan Pau Beltran <joanpau.beltran@socib.cat> 0151 0152 % Copyright (C) 2013-2016 0153 % ICTS SOCIB - Servei d'observacio i prediccio costaner de les Illes Balears 0154 % <http://www.socib.es> 0155 % 0156 % This program is free software: you can redistribute it and/or modify 0157 % it under the terms of the GNU General Public License as published by 0158 % the Free Software Foundation, either version 3 of the License, or 0159 % (at your option) any later version. 0160 % 0161 % This program is distributed in the hope that it will be useful, 0162 % but WITHOUT ANY WARRANTY; without even the implied warranty of 0163 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 0164 % GNU General Public License for more details. 0165 % 0166 % You should have received a copy of the GNU General Public License 0167 % along with this program. If not, see <http://www.gnu.org/licenses/>. 0168 0169 0170 %% Configure toolbox and configuration file path. 0171 glider_toolbox_dir = configGliderToolboxPath(); 0172 glider_toolbox_ver = configGliderToolboxVersion(); 0173 0174 0175 %% Configure external programs paths. 0176 config.wrcprogs = configWRCPrograms(); 0177 0178 0179 %% Configure deployment data paths. 0180 config.paths_public = configDTPathsPublic(); 0181 config.paths_local = configDTPathsLocal(); 0182 0183 0184 %% Configure figure outputs. 0185 [config.figures_processed, config.figures_gridded] = configFigures(); 0186 0187 0188 %% Configure NetCDF outputs. 0189 config.output_netcdf_l0_slocum = configDTOutputNetCDFL0Slocum(); 0190 config.output_netcdf_l0_seaglider = configDTOutputNetCDFL0Seaglider(); 0191 config.output_netcdf_l0_seaexplorer = configDTOutputNetCDFL0SeaExplorer(); 0192 config.output_netcdf_l1 = configDTOutputNetCDFL1(); 0193 config.output_netcdf_l2 = configDTOutputNetCDFL2(); 0194 0195 0196 %% Configure processing options. 0197 config.preprocessing_options_slocum = configDataPreprocessingSlocum(); 0198 config.preprocessing_options_seaglider = configDataPreprocessingSeaglider(); 0199 config.preprocessing_options_seaexplorer = configDataPreprocessingSeaExplorer(); 0200 config.processing_options_slocum_g1 = configDataProcessingSlocumG1(); 0201 config.processing_options_slocum_g2 = configDataProcessingSlocumG2(); 0202 config.processing_options_seaglider = configDataProcessingSeaglider(); 0203 config.processing_options_seaexplorer = configDataProcessingSeaExplorer(); 0204 config.gridding_options = configDataGridding(); 0205 0206 0207 %% Configure file download and conversion and data loading. 0208 config.file_options_slocum = configDTFileOptionsSlocum(); 0209 config.file_options_seaglider = configDTFileOptionsSeaglider(); 0210 config.file_options_seaexplorer = configDTFileOptionsSeaExplorer(); 0211 0212 0213 %% Configure data base deployment information source. 0214 config.db_access = configDBAccess(); 0215 [config.db_query, config.db_fields] = configDTDeploymentInfoQueryDB(); 0216 0217 0218 %% Get list of deployments to process from database. 0219 disp('Querying information of glider deployments...'); 0220 deployment_list = getDeploymentInfoDB( ... 0221 config.db_query, config.db_access.name, ... 0222 'user', config.db_access.user, 'pass', config.db_access.pass, ... 0223 'server', config.db_access.server, 'driver', config.db_access.driver, ... 0224 'fields', config.db_fields); 0225 if isempty(deployment_list) 0226 disp('Selected glider deployments are not available.'); 0227 return 0228 else 0229 disp(['Selected deployments found: ' num2str(numel(deployment_list)) '.']); 0230 end 0231 0232 0233 %% Process active deployments. 0234 for deployment_idx = 1:numel(deployment_list) 0235 %% Set deployment field shortcut variables and initialize other ones. 0236 % Initialization of big data variables may reduce out of memory problems, 0237 % provided memory is properly freed and not fragmented. 0238 disp(['Processing deployment ' num2str(deployment_idx) '...']); 0239 deployment = deployment_list(deployment_idx); 0240 processing_log = strfstruct(config.paths_local.processing_log, deployment); 0241 binary_dir = strfstruct(config.paths_local.binary_path, deployment); 0242 cache_dir = strfstruct(config.paths_local.cache_path, deployment); 0243 log_dir = strfstruct(config.paths_local.log_path, deployment); 0244 ascii_dir = strfstruct(config.paths_local.ascii_path, deployment); 0245 figure_dir = strfstruct(config.paths_local.figure_path, deployment); 0246 netcdf_l0_file = strfstruct(config.paths_local.netcdf_l0, deployment); 0247 netcdf_l1_file = strfstruct(config.paths_local.netcdf_l1, deployment); 0248 netcdf_l2_file = strfstruct(config.paths_local.netcdf_l2, deployment); 0249 source_files = {}; 0250 meta_raw = struct(); 0251 data_raw = struct(); 0252 meta_preprocessed = struct(); 0253 data_preprocessed = struct(); 0254 meta_processed = struct(); 0255 data_processed = struct(); 0256 meta_gridded = struct(); 0257 data_gridded = struct(); 0258 outputs = struct(); 0259 figures = struct(); 0260 deployment_name = deployment.deployment_name; 0261 deployment_id = deployment.deployment_id; 0262 deployment_start = deployment.deployment_start; 0263 deployment_end = deployment.deployment_end; 0264 glider_name = deployment.glider_name; 0265 glider_model = deployment.glider_model; 0266 glider_serial = deployment.glider_serial; 0267 glider_type = ''; 0268 if ~isempty(regexpi(glider_model, '.*slocum.*g1.*', 'match', 'once')) 0269 glider_type = 'slocum_g1'; 0270 elseif ~isempty(regexpi(glider_model, '.*slocum.*g2.*', 'match', 'once')) 0271 glider_type = 'slocum_g2'; 0272 elseif ~isempty(regexpi(glider_model, '.*seaglider.*', 'match', 'once')) 0273 glider_type = 'seaglider'; 0274 elseif ~isempty(regexpi(glider_model, '.*seaexplorer.*', 'match', 'once')) 0275 glider_type = 'seaexplorer'; 0276 end 0277 % Options depending on the type of glider: 0278 switch glider_type 0279 case 'slocum_g1' 0280 file_options = config.file_options_slocum; 0281 preprocessing_options = config.preprocessing_options_slocum; 0282 processing_options = config.processing_options_slocum_g1; 0283 netcdf_l0_options = config.output_netcdf_l0_slocum; 0284 case 'slocum_g2' 0285 file_options = config.file_options_slocum; 0286 preprocessing_options = config.preprocessing_options_slocum; 0287 processing_options = config.processing_options_slocum_g2; 0288 netcdf_l0_options = config.output_netcdf_l0_slocum; 0289 case 'seaglider' 0290 file_options = config.file_options_seaglider; 0291 preprocessing_options = config.preprocessing_options_seaglider; 0292 processing_options = config.processing_options_seaglider; 0293 netcdf_l0_options = config.output_netcdf_l0_seaglider; 0294 case 'seaexplorer' 0295 file_options = config.file_options_seaexplorer; 0296 preprocessing_options = config.preprocessing_options_seaexplorer; 0297 processing_options = config.processing_options_seaexplorer; 0298 netcdf_l0_options = config.output_netcdf_l0_seaexplorer; 0299 end 0300 if isfield(deployment, 'calibrations') 0301 preprocessing_options.calibration_parameter_list = deployment.calibrations; 0302 end 0303 gridding_options = config.gridding_options; 0304 netcdf_l1_options = config.output_netcdf_l1; 0305 netcdf_l2_options = config.output_netcdf_l2; 0306 figproc_options = config.figures_processed; 0307 figgrid_options = config.figures_gridded; 0308 0309 0310 %% Start deployment processing logging. 0311 % DIARY will fail if log file base directory does not exist. 0312 % Create the base directory first, if needed. 0313 % This is an ugly hack (the best known way) to check if the directory exists. 0314 [processing_log_dir, ~, ~] = fileparts(processing_log); 0315 [status, attrout] = fileattrib(processing_log_dir); 0316 if ~status 0317 [status, message] = mkdir(processing_log_dir); 0318 elseif ~attrout.directory 0319 status = false; 0320 message = 'not a directory'; 0321 end 0322 % Enable log only if directory was already there or has been created properly. 0323 if status 0324 try 0325 diary(processing_log); 0326 diary('on'); 0327 catch exception 0328 disp(['Error enabling processing log diary ' processing_log ':']); 0329 disp(getReport(exception, 'extended')); 0330 end 0331 else 0332 disp(['Error creating processing log directory ' processing_log_dir ':']); 0333 disp(message); 0334 end 0335 disp(['Deployment processing start time: ' ... 0336 datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00')]); 0337 0338 0339 %% Report toolbox version: 0340 disp(['Toolbox version: ' glider_toolbox_ver]); 0341 0342 0343 %% Report deployment information. 0344 disp('Deployment information:') 0345 disp([' Glider name : ' glider_name]); 0346 disp([' Glider model : ' glider_model]); 0347 disp([' Glider serial : ' glider_serial]); 0348 disp([' Deployment identifier: ' num2str(deployment_id)]); 0349 disp([' Deployment name : ' deployment_name]); 0350 disp([' Deployment start : ' datestr(deployment_start)]); 0351 if isnan(deployment_end) 0352 disp([' Deployment end : ' 'undefined']); 0353 else 0354 disp([' Deployment end : ' datestr(deployment_end)]); 0355 end 0356 0357 0358 %% Convert binary glider files to ascii human readable format, if needed. 0359 % Check deployment files available in binary directory, 0360 % convert them to ascii format in the ascii directory, 0361 % and store the returned absolute path for later use. 0362 % Since some conversion may fail use a cell array of string cell arrays and 0363 % flatten it when finished, leaving only the succesfully created dbas. 0364 % Give a second try to failing files, because they might have failed due to 0365 % a missing cache file generated later. 0366 switch glider_type 0367 case {'slocum_g1' 'slocum_g2'} 0368 if file_options.format_conversion 0369 % Look for xbds in binary directory. 0370 disp('Converting binary data files to ascii format...'); 0371 bin_dir_contents = dir(binary_dir); 0372 xbd_select = ~[bin_dir_contents.isdir] ... 0373 & ~cellfun(@isempty, regexp({bin_dir_contents.name}, file_options.xbd_name_pattern)); 0374 xbd_names = {bin_dir_contents(xbd_select).name}; 0375 xbd_sizes = [bin_dir_contents(xbd_select).bytes]; 0376 disp(['Binary files found: ' num2str(numel(xbd_names)) ... 0377 ' (' num2str(sum(xbd_sizes)*2^-10) ' kB).']); 0378 new_files = cell(size(xbd_names)); 0379 for conversion_retry = 1:2 0380 for xbd_idx = 1:numel(xbd_names) 0381 if isempty(new_files{xbd_idx}) 0382 xbd_name_ext = xbd_names{xbd_idx}; 0383 dba_name_ext = regexprep(xbd_name_ext, ... 0384 file_options.xbd_name_pattern, ... 0385 file_options.dba_name_replace); 0386 xbd_fullfile = fullfile(binary_dir, xbd_name_ext); 0387 dba_fullfile = fullfile(ascii_dir, dba_name_ext); 0388 try 0389 new_files{xbd_idx} = ... 0390 {xbd2dba(xbd_fullfile, dba_fullfile, 'cache', cache_dir, ... 0391 'cmdname', config.wrcprogs.dbd2asc)}; 0392 catch exception 0393 new_files{xbd_idx} = {}; 0394 if conversion_retry == 2 0395 disp(['Error converting binary file ' xbd_name_ext ':']); 0396 disp(getReport(exception, 'extended')); 0397 end 0398 end 0399 end 0400 end 0401 end 0402 new_files = [new_files{:}]; 0403 disp(['Binary files converted: ' ... 0404 num2str(numel(new_files)) ' of ' num2str(numel(xbd_names)) '.']); 0405 end 0406 otherwise 0407 end 0408 0409 0410 %% Load data from ascii deployment glider files. 0411 disp('Loading raw deployment data from text files...'); 0412 load_start = utc2posixtime(deployment_start); 0413 load_final = posixtime(); 0414 if ~isnan(deployment_end) 0415 load_final = utc2posixtime(deployment_end); 0416 end 0417 try 0418 switch glider_type 0419 case {'slocum_g1' 'slocum_g2'} 0420 [meta_raw, data_raw] = ... 0421 loadSlocumData(ascii_dir, ... 0422 file_options.dba_name_pattern_nav, ... 0423 file_options.dba_name_pattern_sci, ... 0424 'timenav', file_options.dba_time_sensor_nav, ... 0425 'timesci', file_options.dba_time_sensor_sci, ... 0426 'sensors', file_options.dba_sensors, ... 0427 'period', [load_start load_final], ... 0428 'format', 'struct'); 0429 source_files = {meta_raw.headers.filename_label}; 0430 case 'seaglider' 0431 [meta_raw, data_raw] = ... 0432 loadSeagliderData(ascii_dir, ... 0433 file_options.log_name_pattern, ... 0434 file_options.eng_name_pattern, ... 0435 'columns', file_options.eng_columns, ... 0436 'params' , file_options.log_params, ... 0437 'period', [load_start load_final], ... 0438 'format', 'merged'); 0439 source_files = meta_raw.sources; 0440 case {'seaexplorer'} 0441 [meta_raw, data_raw] = ... 0442 loadSeaExplorerData(ascii_dir, ... 0443 file_options.gli_name_pattern, ... 0444 file_options.pld_name_pattern, ... 0445 'timegli', file_options.gli_time, ... 0446 'timepld', file_options.pld_time, ... 0447 'format', 'struct'); 0448 source_files = meta_raw.sources; 0449 otherwise 0450 warning('glider_toolbox:main_glider_data_processing_dt:InvalidGliderType', ... 0451 'Unknown glider model: %s.', glider_model); 0452 end 0453 catch exception 0454 disp('Error loading raw data:'); 0455 disp(getReport(exception, 'extended')); 0456 end 0457 0458 0459 %% Add source files to deployment structure if loading succeeded. 0460 if isempty(source_files) 0461 disp('No deployment data, processing and product generation will be skipped.'); 0462 else 0463 disp(['Files loaded in deployment period: ' num2str(numel(source_files)) '.']); 0464 deployment.source_files = sprintf('%s\n', source_files{:}); 0465 end 0466 0467 0468 %% Generate L0 NetCDF file (raw/preprocessed data), if needed and possible. 0469 if ~isempty(fieldnames(data_raw)) && ~isempty(netcdf_l0_file) 0470 disp('Generating NetCDF L0 output...'); 0471 try 0472 switch glider_type 0473 case {'slocum_g1' 'slocum_g2'} 0474 outputs.netcdf_l0 = generateOutputNetCDF( ... 0475 netcdf_l0_file, data_raw, meta_raw, deployment, ... 0476 netcdf_l0_options.variables, ... 0477 netcdf_l0_options.dimensions, ... 0478 netcdf_l0_options.attributes, ... 0479 'time', {'m_present_time' 'sci_m_present_time'}, ... 0480 'position', {'m_gps_lon' 'm_gps_lat'; 'm_lon' 'm_lat'}, ... 0481 'position_conversion', @nmea2deg, ... 0482 'vertical', {'m_depth' 'sci_water_pressure'}, ... 0483 'vertical_conversion', {[] @(z)(z * 10)}, ... 0484 'vertical_positive', {'down'} ); 0485 case 'seaglider' 0486 outputs.netcdf_l0 = generateOutputNetCDF( ... 0487 netcdf_l0_file, data_raw, meta_raw, deployment, ... 0488 netcdf_l0_options.variables, ... 0489 netcdf_l0_options.dimensions, ... 0490 netcdf_l0_options.attributes, ... 0491 'time', {'elaps_t'}, ... 0492 'time_conversion', @(t)(t + meta_raw.start_secs), ... 0493 'position', {'GPSFIX_fixlon' 'GPSFIX_fixlat'}, ... 0494 'position_conversion', @nmea2deg, ... 0495 'vertical', {'depth'}, ... 0496 'vertical_conversion', {@(z)(z * 10)}, ... 0497 'vertical_positive', {'down'} ); 0498 case {'seaexplorer'} 0499 outputs.netcdf_l0 = generateOutputNetCDF( ... 0500 netcdf_l0_file, data_raw, meta_raw, deployment, ... 0501 netcdf_l0_options.variables, ... 0502 netcdf_l0_options.dimensions, ... 0503 netcdf_l0_options.attributes, ... 0504 'time', {'Timestamp' 'PLD_REALTIMECLOCK'}, ... 0505 'position', {'NAV_LONGITUDE' 'NAV_LATITUDE'; 'Lon' 'Lat'}, ... 0506 'position_conversion', @nmea2deg, ... 0507 'vertical', {'Depth' 'SBD_PRESSURE'}, ... 0508 'vertical_conversion', {[] @(z)(z * 10)}, ... 0509 'vertical_positive', {'down'} ); 0510 end 0511 disp(['Output NetCDF L0 (raw data) generated: ' outputs.netcdf_l0 '.']); 0512 catch exception 0513 disp(['Error generating NetCDF L0 (raw data) output ' netcdf_l0_file ':']); 0514 disp(getReport(exception, 'extended')); 0515 end 0516 end 0517 0518 0519 %% Preprocess raw glider data. 0520 if ~isempty(fieldnames(data_raw)) 0521 disp('Preprocessing raw data...'); 0522 try 0523 switch glider_type 0524 case 'seaglider' 0525 seaglider_time_sensor_select = ... 0526 strcmp('elaps_t', {preprocessing_options.time_list.time}); 0527 preprocessing_options.time_list(seaglider_time_sensor_select).conversion = ... 0528 @(t)(t + meta_raw.start_secs); 0529 end 0530 [data_preprocessed, meta_preprocessed] = ... 0531 preprocessGliderData(data_raw, meta_raw, preprocessing_options); 0532 catch exception 0533 disp('Error preprocessing glider deployment data:'); 0534 disp(getReport(exception, 'extended')); 0535 end 0536 end 0537 0538 0539 %% Process preprocessed glider data. 0540 if ~isempty(fieldnames(data_preprocessed)) 0541 disp('Processing glider data...'); 0542 try 0543 [data_processed, meta_processed] = ... 0544 processGliderData(data_preprocessed, meta_preprocessed, processing_options); 0545 catch exception 0546 disp('Error processing glider deployment data:'); 0547 disp(getReport(exception, 'extended')); 0548 end 0549 end 0550 0551 0552 %% Generate L1 NetCDF file (processed data), if needed and possible. 0553 if ~isempty(fieldnames(data_processed)) && ~isempty(netcdf_l1_file) 0554 disp('Generating NetCDF L1 output...'); 0555 try 0556 outputs.netcdf_l1 = generateOutputNetCDF( ... 0557 netcdf_l1_file, data_processed, meta_processed, deployment, ... 0558 netcdf_l1_options.variables, ... 0559 netcdf_l1_options.dimensions, ... 0560 netcdf_l1_options.attributes); 0561 disp(['Output NetCDF L1 (processed data) generated: ' ... 0562 outputs.netcdf_l1 '.']); 0563 catch exception 0564 disp(['Error generating NetCDF L1 (processed data) output ' ... 0565 netcdf_l1_file ':']); 0566 disp(getReport(exception, 'extended')); 0567 end 0568 end 0569 0570 0571 %% Generate processed data figures. 0572 if ~isempty(fieldnames(data_processed)) && ~isempty(figure_dir) 0573 disp('Generating figures from processed data...'); 0574 try 0575 figures.figproc = generateGliderFigures( ... 0576 data_processed, figproc_options, ... 0577 'date', datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00'), ... 0578 'dirname', figure_dir); 0579 catch exception 0580 disp('Error generating processed data figures:'); 0581 disp(getReport(exception, 'extended')); 0582 end 0583 end 0584 0585 0586 %% Grid processed glider data. 0587 if ~isempty(fieldnames(data_processed)) 0588 disp('Gridding glider data...'); 0589 try 0590 [data_gridded, meta_gridded] = ... 0591 gridGliderData(data_processed, meta_processed, gridding_options); 0592 catch exception 0593 disp('Error gridding glider deployment data:'); 0594 disp(getReport(exception, 'extended')); 0595 end 0596 end 0597 0598 0599 %% Generate L2 (gridded data) netcdf file, if needed and possible. 0600 if ~isempty(fieldnames(data_gridded)) && ~isempty(netcdf_l2_file) 0601 disp('Generating NetCDF L2 output...'); 0602 try 0603 outputs.netcdf_l2 = generateOutputNetCDF( ... 0604 netcdf_l2_file, data_gridded, meta_gridded, deployment, ... 0605 netcdf_l2_options.variables, ... 0606 netcdf_l2_options.dimensions, ... 0607 netcdf_l2_options.attributes); 0608 disp(['Output NetCDF L2 (gridded data) generated: ' ... 0609 outputs.netcdf_l2 '.']); 0610 catch exception 0611 disp(['Error generating NetCDF L2 (gridded data) output ' ... 0612 netcdf_l2_file ':']); 0613 disp(getReport(exception, 'extended')); 0614 end 0615 end 0616 0617 0618 %% Generate gridded data figures. 0619 if ~isempty(fieldnames(data_gridded)) && ~isempty(figure_dir) 0620 disp('Generating figures from gridded data...'); 0621 try 0622 figures.figgrid = generateGliderFigures( ... 0623 data_gridded, figgrid_options, ... 0624 'date', datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00'), ... 0625 'dirname', figure_dir); 0626 catch exception 0627 disp('Error generating gridded data figures:'); 0628 disp(getReport(exception, 'extended')); 0629 end 0630 end 0631 0632 0633 %% Copy selected products to corresponding public location, if needed. 0634 if ~isempty(fieldnames(outputs)) 0635 disp('Copying public outputs...'); 0636 output_name_list = fieldnames(outputs); 0637 for output_name_idx = 1:numel(output_name_list) 0638 output_name = output_name_list{output_name_idx}; 0639 if isfield(config.paths_public, output_name) ... 0640 && ~isempty(config.paths_public.(output_name)) 0641 output_local_file = outputs.(output_name); 0642 output_public_file = ... 0643 strfstruct(config.paths_public.(output_name), deployment); 0644 output_public_dir = fileparts(output_public_file); 0645 [status, attrout] = fileattrib(output_public_dir); 0646 if ~status 0647 [status, message] = mkdir(output_public_dir); 0648 elseif ~attrout.directory 0649 status = false; 0650 message = 'not a directory'; 0651 end 0652 if status 0653 [success, message] = copyfile(output_local_file, output_public_file); 0654 if success 0655 disp(['Public output ' output_name ' succesfully copied: ' ... 0656 output_public_file '.']); 0657 else 0658 disp(['Error creating public copy of deployment product ' ... 0659 output_name ': ' output_public_file '.']); 0660 disp(message); 0661 end 0662 else 0663 disp(['Error creating public output directory ' ... 0664 output_public_dir ':']); 0665 disp(message); 0666 end 0667 end 0668 end 0669 end 0670 0671 0672 %% Copy selected figures to its public location, if needed. 0673 % Copy all generated figures or only the ones in the include list (if any) 0674 % excluding the ones in the exclude list. 0675 if ~isempty(fieldnames(figures)) ... 0676 && isfield(config.paths_public, 'figure_dir') ... 0677 && ~isempty(config.paths_public.figure_dir) 0678 disp('Copying public figures...'); 0679 public_figure_baseurl = ... 0680 strfstruct(config.paths_public.figure_url, deployment); 0681 public_figure_dir = ... 0682 strfstruct(config.paths_public.figure_dir, deployment); 0683 public_figure_include_all = true; 0684 public_figure_exclude_none = true; 0685 public_figure_include_list = []; 0686 public_figure_exclude_list = []; 0687 if isfield(config.paths_public, 'figure_include') 0688 public_figure_include_all = false; 0689 public_figure_include_list = config.paths_public.figure_include; 0690 end 0691 if isfield(config.paths_public, 'figure_exclude') 0692 public_figure_exclude_none = false; 0693 public_figure_exclude_list = config.paths_public.figure_exclude; 0694 end 0695 public_figures = struct(); 0696 public_figures_local = struct(); 0697 figure_output_name_list = fieldnames(figures); 0698 for figure_output_name_idx = 1:numel(figure_output_name_list) 0699 figure_output_name = figure_output_name_list{figure_output_name_idx}; 0700 figure_output = figures.(figure_output_name); 0701 figure_name_list = fieldnames(figure_output); 0702 for figure_name_idx = 1:numel(figure_name_list) 0703 figure_name = figure_name_list{figure_name_idx}; 0704 if (public_figure_include_all ... 0705 || ismember(figure_name, public_figure_include_list)) ... 0706 && (public_figure_exclude_none ... 0707 || ~ismember(figure_name, public_figure_exclude_list)) 0708 if isfield(public_figures_local, figure_name) 0709 disp(['Warning: figure ' figure_name ' appears to be duplicated.']); 0710 else 0711 public_figures_local.(figure_name) = figure_output.(figure_name); 0712 end 0713 end 0714 end 0715 end 0716 public_figure_name_list = fieldnames(public_figures_local); 0717 if ~isempty(public_figure_name_list) 0718 [status, attrout] = fileattrib(public_figure_dir); 0719 if ~status 0720 [status, message] = mkdir(public_figure_dir); 0721 elseif ~attrout.directory 0722 status = false; 0723 message = 'not a directory'; 0724 end 0725 if status 0726 for public_figure_name_idx = 1:numel(public_figure_name_list) 0727 public_figure_name = public_figure_name_list{public_figure_name_idx}; 0728 figure_local = public_figures_local.(public_figure_name); 0729 figure_public = figure_local; 0730 figure_public.url = ... 0731 [public_figure_baseurl '/' ... 0732 figure_public.filename '.' figure_public.format]; 0733 figure_public.dirname = public_figure_dir; 0734 figure_public.fullfile = ... 0735 fullfile(figure_public.dirname, ... 0736 [figure_public.filename '.' figure_public.format]); 0737 [success, message] = ... 0738 copyfile(figure_local.fullfile, figure_public.fullfile); 0739 if success 0740 public_figures.(public_figure_name) = figure_public; 0741 disp(['Public figure ' public_figure_name ' succesfully copied.']); 0742 else 0743 disp(['Error creating public copy of figure ' ... 0744 public_figure_name ': ' figure_public.fullfile '.']); 0745 disp(message); 0746 end 0747 end 0748 else 0749 disp(['Error creating public figure directory ' public_figure_dir ':']); 0750 disp(message); 0751 end 0752 end 0753 % Write the figure information to the JSON service file. 0754 if isfield(config.paths_public, 'figure_info') ... 0755 && ~isempty(config.paths_public.figure_info) 0756 disp('Generating figure information service file...'); 0757 public_figure_info_file = ... 0758 strfstruct(config.paths_public.figure_info, deployment); 0759 try 0760 savejson(public_figures, public_figure_info_file); 0761 disp(['Figure information service file successfully generated: ' ... 0762 public_figure_info_file]); 0763 catch exception 0764 disp(['Error creating figure information service file ' ... 0765 public_figure_info_file ':']); 0766 disp(message); 0767 end 0768 end 0769 end 0770 0771 0772 %% Stop deployment processing logging. 0773 disp(['Deployment processing end time: ' ... 0774 datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00')]); 0775 diary('off'); 0776 0777 end