MAIN_GLIDER_DATA_PROCESSING_RT Run near real time glider processing chain.
Description:
This script develops the full processing chain for real time glider data:
- Check for active deployments from deployment information source.
- Download new or updated deployment raw data files.
- Convert downloaded files to human readable format.
- Load data from all files in a single and consistent structure.
- Generate standarized product version of raw data (NetCDF level 0).
- Preprocess raw data applying simple unit conversions and factory
calibrations without modifying their nominal value:
-- Select reference sensors for time and space coordinates.
Perform unit conversions if necessary.
-- Select extra navigation sensors: waypoints, pitch, depth...
Perform unit conversions if necessary.
-- Select sensors of interest: CTD, oxygen, ocean color...
Perform unit conversions and factory calibrations if necessary.
- Process preprocessed data to obtain well referenced trajectory data
with new derived measurements and corrections:
-- Fill missing values of time and space reference sensors.
-- Fill missing values of other navigation sensors.
-- Identify transect boundaries at waypoint changes.
-- Identify cast boundaries from vertical direction changes.
-- Apply generic sensor processings: sensor lag correction...
-- Process CTD data: pressure filtering, thermal lag correction...
-- Derive new measurements: depth, salinity, density...
- Generate standarized product version of trajectory data (NetCDF
level 1).
- Generate descriptive figures from trajectory data.
- Interpolate/bin trajectory data to obtain gridded data (vertical
instantaneous profiles of already processed data).
- Generate standarized product version of gridded data (NetCDF level 2).
- Generate descriptive figures from gridded data.
- Copy generated data products to its public location, if needed.
- Copy generated figures to its public location and generate figure
information service file, if needed.
Deployment information is queried from a data base by GETDEPLOYMENTINFODB.
Data base access parameters may be configured in CONFIGDBACCESS.
Selected deployments and their metadata fields may be configured in
CONFIGRTDEPLOYMENTINFOQUERYDB.
For each deployment, the messages produced during each processing step are
recorded to a log file. This recording is enabled just before the start of
the processing of the deployment, and it is turned off when the processing
finishes, with the function DIARY.
New raw data files of the deployment are fetched from remote servers.
For Slocum gliders, binary and log files are retrieved by
GETDOCKSERVERFILES from each dockserver specified in CONFIGDOCKSERVERS,
and stored in the binary and log directories configured in
CONFIGRTPATHSLOCAL. For Seaglider gliders, engineering data files and
log data files are retrieved by GETBASESTATIONFILES from each basestation
specified in CONFIGBASESTATIONS, and stored in the ascii folder specified
in CONFIGRTPATHSLOCAL. For SeaExplorer gliders the file retrieval is not
implemented yet. The names of the files to download may be restricted in
CONFIGRTFILEOPTIONSSLOCUM and CONFIGRTFILEOPTIONSSEAGLIDER.
For Slocum gliders fetched binary files are converted to text format.
The conversion is performed by function XBD2DBA, which is called for each
binary file with a renaming pattern to specify the name of the resulting
text file, and performs a system call to program 'dbd2asc' by WRC.
The path to the 'dbd2asc' program may be configured in CONFIGWRCPROGRAMS.
File conversion options may be configured in CONFIGRTFILEOPTIONSSLOCUM,
and the directory for converted text files in CONFIGRTPATHSLOCAL.
Input deployment raw data is loaded from the directory of raw text files
with LOADSLOCUMDATA, LOADSEAGLIDERDATA or LOADSEAEXPLORERDATA.
Data loading options may be configured in CONFIGRTFILEOPTIONSSLOCUM,
CONFIGRTFILEOPTIONSSEAGLIDER, and CONFIGRTFILEOPTIONSSEAEXPLORER.
Output products, figures and processing logs are generated to local paths.
Input and output paths may be configured using expressions built upon
deployment field value replacements in CONFIGRTPATHSLOCAL.
Raw data is preprocessed to apply some simple unit conversions with the
function PREPROCESSGLIDERDATA. The preprocessing options and its
parameters may be configured in CONFIGDATAPREPROCESSINGSLOCUM,
CONFIGDATAPREPROCESSINGSEAGLIDER and CONFIGDATAPREPROCESSINGSEAEXPLORER.
Preprocessed data is processed with PROCESSGLIDERDATA to obtain properly
referenced data with a trajectory data structure. The desired processing
actions (interpolations, filterings, corrections and derivations)
and its parameters may be configured in CONFIGDATAPROCESSINGSLOCUMG1,
CONFIGDATAPROCESSINGSLOCUMG2, CONFIGDATAPROCESSINGSEAGLIDER and
CONFIGDATAPROCESSINGSEAEXPLORER.
Processed data is interpolated/binned with GRIDGLIDERDATA to obtain a data
set with the structure of a trajectory of instantaneous vertical profiles
sampled at a common set of regular depth levels. The desired gridding
parameters may be configured in CONFIGDATAGRIDDING.
Standard products in NetCDF format are generated from raw data,
processed data and gridded data with GENERATEOUTPUTNETCDF.
Raw data is stored in NetCDF format as level 0 output product.
This file mimics the appearance of the raw data text files, but gathering
all useful data in a single place. Hence, the structure of the resulting
NetCDF file varies with each type of glider, and may be configured
in CONFIGRTOUTPUTNETCDFL0SLOCUM, CONFIGRTOUTPUTNETCDFL0SEAGLIDER and
CONFIGRTOUTPUTNETCDFL0SEAEXPLORER. Processed and gridded data sets are
stored in NetCDF format as level 1 and level 2 output products
respectively. The structure of these files does not depend on the type
of glider the data comes from, and may be configured in
CONFIGRTOUTPUNETCDFL1 and CONFIGRTOUTPUTNETCDFL2 respectively.
Figures describing the collected glider data may be generated from
processed data and from gridded data. Figures are generated by
GENERATEGLIDERFIGURES, and may be configured in CONFIGFIGURES.
Available plots are: scatter plots of measurements on vertical transect
sections, temperature-salinity diagrams, trajectory and current maps,
and profile statistics plots. Other plot functions may be used,
provided that their call syntax is compatible with GENERATEGLIDERFIGURES.
Selected data output products and figures may be copied to a public
location for distribution purposes. For figures, a service file describing
the available figures and their public location may also be generated.
This file is generated by function SAVEJSON with the figure information
returned by GENERATEGLIDERFIGURES updated with the new public location.
Public products and figures to copy and their locations may be configured
in CONFIGRTPATHSPUBLIC.
See also:
CONFIGWRCPROGRAMS
CONFIGDOCKSERVERS
CONFIGBASESTATIONS
CONFIGDBACCESS
CONFIGRTDEPLOYMENTINFOQUERYDB
CONFIGRTPATHSLOCAL
CONFIGRTFILEOPTIONSSLOCUM
CONFIGRTFILEOPTIONSSEAGLIDER
CONFIGRTFILEOPTIONSSEAEXPLORER
CONFIGDATAPREPROCESSINGSLOCUM
CONFIGDATAPREPROCESSINGSEAGLIDER
CONFIGDATAPREPROCESSINGSEAEXPLORER
CONFIGDATAPROCESSINGSLOCUMG1
CONFIGDATAPROCESSINGSLOCUMG2
CONFIGDATAPROCESSINGSEAGLIDER
CONFIGDATAPROCESSINGSEAEXPLORER
CONFIGDATAGRIDDING
CONFIGRTOUTPUTNETCDFL0SLOCUM
CONFIGRTOUTPUTNETCDFL0SEAGLIDER
CONFIGRTOUTPUTNETCDFL0SEAEXPLORER
CONFIGRTOUTPUTNETCDFL1
CONFIGRTOUTPUTNETCDFL2
CONFIGFIGURES
GETDEPLOYMENTINFODB
GETDOCKSERVERFILES
GETBASESTATIONFILES
LOADSLOCUMDATA
PREPROCESSGLIDERDATA
PROCESSGLIDERDATA
GRIDGLIDERDATA
GENERATEOUTPUTNETCDF
GENERATEFIGURES
DIARY
STRFSTRUCT
XBD2DBA
SAVEJSON
Notes:
This script is based on the previous work by Tomeu Garau. He is the true
glider man.
Authors:
Joan Pau Beltran <joanpau.beltran@socib.cat>main_glider_data_processing_rt.m
0001 %MAIN_GLIDER_DATA_PROCESSING_RT Run near real time glider processing chain. 0002 % 0003 % Description: 0004 % This script develops the full processing chain for real time glider data: 0005 % - Check for active deployments from deployment information source. 0006 % - Download new or updated deployment raw data files. 0007 % - Convert downloaded files to human readable format. 0008 % - Load data from all files in a single and consistent structure. 0009 % - Generate standarized product version of raw data (NetCDF level 0). 0010 % - Preprocess raw data applying simple unit conversions and factory 0011 % calibrations without modifying their nominal value: 0012 % -- Select reference sensors for time and space coordinates. 0013 % Perform unit conversions if necessary. 0014 % -- Select extra navigation sensors: waypoints, pitch, depth... 0015 % Perform unit conversions if necessary. 0016 % -- Select sensors of interest: CTD, oxygen, ocean color... 0017 % Perform unit conversions and factory calibrations if necessary. 0018 % - Process preprocessed data to obtain well referenced trajectory data 0019 % with new derived measurements and corrections: 0020 % -- Fill missing values of time and space reference sensors. 0021 % -- Fill missing values of other navigation sensors. 0022 % -- Identify transect boundaries at waypoint changes. 0023 % -- Identify cast boundaries from vertical direction changes. 0024 % -- Apply generic sensor processings: sensor lag correction... 0025 % -- Process CTD data: pressure filtering, thermal lag correction... 0026 % -- Derive new measurements: depth, salinity, density... 0027 % - Generate standarized product version of trajectory data (NetCDF 0028 % level 1). 0029 % - Generate descriptive figures from trajectory data. 0030 % - Interpolate/bin trajectory data to obtain gridded data (vertical 0031 % instantaneous profiles of already processed data). 0032 % - Generate standarized product version of gridded data (NetCDF level 2). 0033 % - Generate descriptive figures from gridded data. 0034 % - Copy generated data products to its public location, if needed. 0035 % - Copy generated figures to its public location and generate figure 0036 % information service file, if needed. 0037 % 0038 % Deployment information is queried from a data base by GETDEPLOYMENTINFODB. 0039 % Data base access parameters may be configured in CONFIGDBACCESS. 0040 % Selected deployments and their metadata fields may be configured in 0041 % CONFIGRTDEPLOYMENTINFOQUERYDB. 0042 % 0043 % For each deployment, the messages produced during each processing step are 0044 % recorded to a log file. This recording is enabled just before the start of 0045 % the processing of the deployment, and it is turned off when the processing 0046 % finishes, with the function DIARY. 0047 % 0048 % New raw data files of the deployment are fetched from remote servers. 0049 % For Slocum gliders, binary and log files are retrieved by 0050 % GETDOCKSERVERFILES from each dockserver specified in CONFIGDOCKSERVERS, 0051 % and stored in the binary and log directories configured in 0052 % CONFIGRTPATHSLOCAL. For Seaglider gliders, engineering data files and 0053 % log data files are retrieved by GETBASESTATIONFILES from each basestation 0054 % specified in CONFIGBASESTATIONS, and stored in the ascii folder specified 0055 % in CONFIGRTPATHSLOCAL. For SeaExplorer gliders the file retrieval is not 0056 % implemented yet. The names of the files to download may be restricted in 0057 % CONFIGRTFILEOPTIONSSLOCUM and CONFIGRTFILEOPTIONSSEAGLIDER. 0058 % 0059 % For Slocum gliders fetched binary files are converted to text format. 0060 % The conversion is performed by function XBD2DBA, which is called for each 0061 % binary file with a renaming pattern to specify the name of the resulting 0062 % text file, and performs a system call to program 'dbd2asc' by WRC. 0063 % The path to the 'dbd2asc' program may be configured in CONFIGWRCPROGRAMS. 0064 % File conversion options may be configured in CONFIGRTFILEOPTIONSSLOCUM, 0065 % and the directory for converted text files in CONFIGRTPATHSLOCAL. 0066 % 0067 % Input deployment raw data is loaded from the directory of raw text files 0068 % with LOADSLOCUMDATA, LOADSEAGLIDERDATA or LOADSEAEXPLORERDATA. 0069 % Data loading options may be configured in CONFIGRTFILEOPTIONSSLOCUM, 0070 % CONFIGRTFILEOPTIONSSEAGLIDER, and CONFIGRTFILEOPTIONSSEAEXPLORER. 0071 % 0072 % Output products, figures and processing logs are generated to local paths. 0073 % Input and output paths may be configured using expressions built upon 0074 % deployment field value replacements in CONFIGRTPATHSLOCAL. 0075 % 0076 % Raw data is preprocessed to apply some simple unit conversions with the 0077 % function PREPROCESSGLIDERDATA. The preprocessing options and its 0078 % parameters may be configured in CONFIGDATAPREPROCESSINGSLOCUM, 0079 % CONFIGDATAPREPROCESSINGSEAGLIDER and CONFIGDATAPREPROCESSINGSEAEXPLORER. 0080 % 0081 % Preprocessed data is processed with PROCESSGLIDERDATA to obtain properly 0082 % referenced data with a trajectory data structure. The desired processing 0083 % actions (interpolations, filterings, corrections and derivations) 0084 % and its parameters may be configured in CONFIGDATAPROCESSINGSLOCUMG1, 0085 % CONFIGDATAPROCESSINGSLOCUMG2, CONFIGDATAPROCESSINGSEAGLIDER and 0086 % CONFIGDATAPROCESSINGSEAEXPLORER. 0087 % 0088 % Processed data is interpolated/binned with GRIDGLIDERDATA to obtain a data 0089 % set with the structure of a trajectory of instantaneous vertical profiles 0090 % sampled at a common set of regular depth levels. The desired gridding 0091 % parameters may be configured in CONFIGDATAGRIDDING. 0092 % 0093 % Standard products in NetCDF format are generated from raw data, 0094 % processed data and gridded data with GENERATEOUTPUTNETCDF. 0095 % Raw data is stored in NetCDF format as level 0 output product. 0096 % This file mimics the appearance of the raw data text files, but gathering 0097 % all useful data in a single place. Hence, the structure of the resulting 0098 % NetCDF file varies with each type of glider, and may be configured 0099 % in CONFIGRTOUTPUTNETCDFL0SLOCUM, CONFIGRTOUTPUTNETCDFL0SEAGLIDER and 0100 % CONFIGRTOUTPUTNETCDFL0SEAEXPLORER. Processed and gridded data sets are 0101 % stored in NetCDF format as level 1 and level 2 output products 0102 % respectively. The structure of these files does not depend on the type 0103 % of glider the data comes from, and may be configured in 0104 % CONFIGRTOUTPUNETCDFL1 and CONFIGRTOUTPUTNETCDFL2 respectively. 0105 % 0106 % Figures describing the collected glider data may be generated from 0107 % processed data and from gridded data. Figures are generated by 0108 % GENERATEGLIDERFIGURES, and may be configured in CONFIGFIGURES. 0109 % Available plots are: scatter plots of measurements on vertical transect 0110 % sections, temperature-salinity diagrams, trajectory and current maps, 0111 % and profile statistics plots. Other plot functions may be used, 0112 % provided that their call syntax is compatible with GENERATEGLIDERFIGURES. 0113 % 0114 % Selected data output products and figures may be copied to a public 0115 % location for distribution purposes. For figures, a service file describing 0116 % the available figures and their public location may also be generated. 0117 % This file is generated by function SAVEJSON with the figure information 0118 % returned by GENERATEGLIDERFIGURES updated with the new public location. 0119 % Public products and figures to copy and their locations may be configured 0120 % in CONFIGRTPATHSPUBLIC. 0121 % 0122 % See also: 0123 % CONFIGWRCPROGRAMS 0124 % CONFIGDOCKSERVERS 0125 % CONFIGBASESTATIONS 0126 % CONFIGDBACCESS 0127 % CONFIGRTDEPLOYMENTINFOQUERYDB 0128 % CONFIGRTPATHSLOCAL 0129 % CONFIGRTFILEOPTIONSSLOCUM 0130 % CONFIGRTFILEOPTIONSSEAGLIDER 0131 % CONFIGRTFILEOPTIONSSEAEXPLORER 0132 % CONFIGDATAPREPROCESSINGSLOCUM 0133 % CONFIGDATAPREPROCESSINGSEAGLIDER 0134 % CONFIGDATAPREPROCESSINGSEAEXPLORER 0135 % CONFIGDATAPROCESSINGSLOCUMG1 0136 % CONFIGDATAPROCESSINGSLOCUMG2 0137 % CONFIGDATAPROCESSINGSEAGLIDER 0138 % CONFIGDATAPROCESSINGSEAEXPLORER 0139 % CONFIGDATAGRIDDING 0140 % CONFIGRTOUTPUTNETCDFL0SLOCUM 0141 % CONFIGRTOUTPUTNETCDFL0SEAGLIDER 0142 % CONFIGRTOUTPUTNETCDFL0SEAEXPLORER 0143 % CONFIGRTOUTPUTNETCDFL1 0144 % CONFIGRTOUTPUTNETCDFL2 0145 % CONFIGFIGURES 0146 % GETDEPLOYMENTINFODB 0147 % GETDOCKSERVERFILES 0148 % GETBASESTATIONFILES 0149 % LOADSLOCUMDATA 0150 % PREPROCESSGLIDERDATA 0151 % PROCESSGLIDERDATA 0152 % GRIDGLIDERDATA 0153 % GENERATEOUTPUTNETCDF 0154 % GENERATEFIGURES 0155 % DIARY 0156 % STRFSTRUCT 0157 % XBD2DBA 0158 % SAVEJSON 0159 % 0160 % Notes: 0161 % This script is based on the previous work by Tomeu Garau. He is the true 0162 % glider man. 0163 % 0164 % Authors: 0165 % Joan Pau Beltran <joanpau.beltran@socib.cat> 0166 0167 % Copyright (C) 2013-2016 0168 % ICTS SOCIB - Servei d'observacio i prediccio costaner de les Illes Balears 0169 % <http://www.socib.es> 0170 % 0171 % This program is free software: you can redistribute it and/or modify 0172 % it under the terms of the GNU General Public License as published by 0173 % the Free Software Foundation, either version 3 of the License, or 0174 % (at your option) any later version. 0175 % 0176 % This program is distributed in the hope that it will be useful, 0177 % but WITHOUT ANY WARRANTY; without even the implied warranty of 0178 % MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 0179 % GNU General Public License for more details. 0180 % 0181 % You should have received a copy of the GNU General Public License 0182 % along with this program. If not, see <http://www.gnu.org/licenses/>. 0183 0184 0185 %% Configure toolbox and configuration file path. 0186 glider_toolbox_dir = configGliderToolboxPath(); 0187 glider_toolbox_ver = configGliderToolboxVersion(); 0188 0189 0190 %% Configure external programs paths. 0191 config.wrcprogs = configWRCPrograms(); 0192 0193 0194 %% Configure deployment data paths. 0195 config.paths_public = configRTPathsPublic(); 0196 config.paths_local = configRTPathsLocal(); 0197 0198 0199 %% Configure figure outputs. 0200 [config.figures_processed, config.figures_gridded] = configFigures(); 0201 0202 0203 %% Configure NetCDF outputs. 0204 config.output_netcdf_l0_slocum = configRTOutputNetCDFL0Slocum(); 0205 config.output_netcdf_l0_seaglider = configRTOutputNetCDFL0Seaglider(); 0206 config.output_netcdf_l0_seaexplorer = configRTOutputNetCDFL0SeaExplorer(); 0207 config.output_netcdf_l1 = configRTOutputNetCDFL1(); 0208 config.output_netcdf_l2 = configRTOutputNetCDFL2(); 0209 0210 0211 %% Configure processing options. 0212 config.preprocessing_options_slocum = configDataPreprocessingSlocum(); 0213 config.preprocessing_options_seaglider = configDataPreprocessingSeaglider(); 0214 config.preprocessing_options_seaexplorer = configDataPreprocessingSeaExplorer(); 0215 config.processing_options_slocum_g1 = configDataProcessingSlocumG1(); 0216 config.processing_options_slocum_g2 = configDataProcessingSlocumG2(); 0217 config.processing_options_seaglider = configDataProcessingSeaglider(); 0218 config.processing_options_seaexplorer = configDataProcessingSeaExplorer(); 0219 config.gridding_options = configDataGridding(); 0220 0221 0222 %% Configure file download and conversion and data loading. 0223 config.file_options_slocum = configRTFileOptionsSlocum(); 0224 config.file_options_seaglider = configRTFileOptionsSeaglider(); 0225 config.file_options_seaexplorer = configRTFileOptionsSeaExplorer(); 0226 0227 0228 %% Configure dockserver and basestation glider data sources. 0229 config.dockservers = configDockservers(); 0230 config.basestations = configBasestations(); 0231 0232 0233 %% Configure data base deployment information source. 0234 config.db_access = configDBAccess(); 0235 [config.db_query, config.db_fields] = configRTDeploymentInfoQueryDB(); 0236 0237 0238 %% Get list of deployments to process from database. 0239 disp('Querying information of glider deployments...'); 0240 deployment_list = getDeploymentInfoDB( ... 0241 config.db_query, config.db_access.name, ... 0242 'user', config.db_access.user, 'pass', config.db_access.pass, ... 0243 'server', config.db_access.server, 'driver', config.db_access.driver, ... 0244 'fields', config.db_fields); 0245 if isempty(deployment_list) 0246 disp('No active glider deployments available.'); 0247 return 0248 else 0249 disp(['Active deployments found: ' num2str(numel(deployment_list)) '.']); 0250 end 0251 0252 0253 %% Process active deployments. 0254 for deployment_idx = 1:numel(deployment_list) 0255 %% Set deployment field shortcut variables and initialize other ones. 0256 % Initialization of big data variables may reduce out of memory problems, 0257 % provided memory is properly freed and not fragmented. 0258 disp(['Processing deployment ' num2str(deployment_idx) '...']); 0259 deployment = deployment_list(deployment_idx); 0260 processing_log = strfstruct(config.paths_local.processing_log, deployment); 0261 binary_dir = strfstruct(config.paths_local.binary_path, deployment); 0262 cache_dir = strfstruct(config.paths_local.cache_path, deployment); 0263 log_dir = strfstruct(config.paths_local.log_path, deployment); 0264 ascii_dir = strfstruct(config.paths_local.ascii_path, deployment); 0265 figure_dir = strfstruct(config.paths_local.figure_path, deployment); 0266 netcdf_l0_file = strfstruct(config.paths_local.netcdf_l0, deployment); 0267 netcdf_l1_file = strfstruct(config.paths_local.netcdf_l1, deployment); 0268 netcdf_l2_file = strfstruct(config.paths_local.netcdf_l2, deployment); 0269 source_files = {}; 0270 meta_raw = struct(); 0271 data_raw = struct(); 0272 meta_preprocessed = struct(); 0273 data_preprocessed = struct(); 0274 meta_processed = struct(); 0275 data_processed = struct(); 0276 meta_gridded = struct(); 0277 data_gridded = struct(); 0278 outputs = struct(); 0279 figures = struct(); 0280 deployment_name = deployment.deployment_name; 0281 deployment_id = deployment.deployment_id; 0282 deployment_start = deployment.deployment_start; 0283 deployment_end = deployment.deployment_end; 0284 glider_name = deployment.glider_name; 0285 glider_model = deployment.glider_model; 0286 glider_serial = deployment.glider_serial; 0287 glider_type = ''; 0288 if ~isempty(regexpi(glider_model, '.*slocum.*g1.*', 'match', 'once')) 0289 glider_type = 'slocum_g1'; 0290 elseif ~isempty(regexpi(glider_model, '.*slocum.*g2.*', 'match', 'once')) 0291 glider_type = 'slocum_g2'; 0292 elseif ~isempty(regexpi(glider_model, '.*seaglider.*', 'match', 'once')) 0293 glider_type = 'seaglider'; 0294 elseif ~isempty(regexpi(glider_model, '.*seaexplorer.*', 'match', 'once')) 0295 glider_type = 'seaexplorer'; 0296 end 0297 % Options depending on the type of glider: 0298 switch glider_type 0299 case 'slocum_g1' 0300 file_options = config.file_options_slocum; 0301 preprocessing_options = config.preprocessing_options_slocum; 0302 processing_options = config.processing_options_slocum_g1; 0303 netcdf_l0_options = config.output_netcdf_l0_slocum; 0304 case 'slocum_g2' 0305 file_options = config.file_options_slocum; 0306 preprocessing_options = config.preprocessing_options_slocum; 0307 processing_options = config.processing_options_slocum_g2; 0308 netcdf_l0_options = config.output_netcdf_l0_slocum; 0309 case 'seaglider' 0310 file_options = config.file_options_seaglider; 0311 preprocessing_options = config.preprocessing_options_seaglider; 0312 processing_options = config.processing_options_seaglider; 0313 netcdf_l0_options = config.output_netcdf_l0_seaglider; 0314 case 'seaexplorer' 0315 file_options = config.file_options_seaexplorer; 0316 preprocessing_options = config.preprocessing_options_seaexplorer; 0317 processing_options = config.processing_options_seaexplorer; 0318 netcdf_l0_options = config.output_netcdf_l0_seaexplorer; 0319 end 0320 if isfield(deployment, 'calibrations') 0321 preprocessing_options.calibration_parameter_list = deployment.calibrations; 0322 end 0323 gridding_options = config.gridding_options; 0324 netcdf_l1_options = config.output_netcdf_l1; 0325 netcdf_l2_options = config.output_netcdf_l2; 0326 figproc_options = config.figures_processed; 0327 figgrid_options = config.figures_gridded; 0328 0329 0330 %% Start deployment processing logging. 0331 % DIARY will fail if log file base directory does not exist. 0332 % Create the base directory first, if needed. 0333 % This is an ugly hack (the best known way) to check if the directory exists. 0334 [processing_log_dir, ~, ~] = fileparts(processing_log); 0335 [status, attrout] = fileattrib(processing_log_dir); 0336 if ~status 0337 [status, message] = mkdir(processing_log_dir); 0338 elseif ~attrout.directory 0339 status = false; 0340 message = 'not a directory'; 0341 end 0342 % Enable log only if directory was already there or has been created properly. 0343 if status 0344 try 0345 diary(processing_log); 0346 diary('on'); 0347 catch exception 0348 disp(['Error enabling processing log diary ' processing_log ':']); 0349 disp(getReport(exception, 'extended')); 0350 end 0351 else 0352 disp(['Error creating processing log directory ' processing_log_dir ':']); 0353 disp(message); 0354 end 0355 disp(['Deployment processing start time: ' ... 0356 datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00')]); 0357 0358 0359 %% Report toolbox version: 0360 disp(['Toolbox version: ' glider_toolbox_ver]); 0361 0362 0363 %% Report deployment information. 0364 disp('Deployment information:') 0365 disp([' Glider name : ' glider_name]); 0366 disp([' Glider model : ' glider_model]); 0367 disp([' Glider serial : ' glider_serial]); 0368 disp([' Deployment identifier: ' num2str(deployment_id)]); 0369 disp([' Deployment name : ' deployment_name]); 0370 disp([' Deployment start : ' datestr(deployment_start)]); 0371 if isnan(deployment_end) 0372 disp([' Deployment end : ' 'undefined']); 0373 else 0374 disp([' Deployment end : ' datestr(deployment_end)]); 0375 end 0376 0377 0378 %% Download deployment glider files from station(s). 0379 % Check for new or updated deployment files in every dockserver. 0380 % Deployment start time must be truncated to days because the date of 0381 % a binary file is deduced from its name only up to day precission. 0382 % Deployment end time may be undefined. 0383 disp('Download deployment new data...'); 0384 download_start = datenum(datestr(deployment_start,'yyyy-mm-dd'),'yyyy-mm-dd'); 0385 if isnan(deployment_end) 0386 download_final = posixtime2utc(posixtime()); 0387 else 0388 download_final = deployment_end; 0389 end 0390 switch glider_type 0391 case {'slocum_g1' 'slocum_g2'} 0392 new_xbds = cell(size(config.dockservers)); 0393 new_logs = cell(size(config.dockservers)); 0394 for dockserver_idx = 1:numel(config.dockservers) 0395 dockserver = config.dockservers(dockserver_idx); 0396 try 0397 [new_xbds{dockserver_idx}, new_logs{dockserver_idx}] = ... 0398 getDockserverFiles(dockserver, glider_name, binary_dir, log_dir, ... 0399 'xbd', file_options.xbd_name_pattern, ... 0400 'log', file_options.log_name_pattern, ... 0401 'start', download_start, ... 0402 'final', download_final); 0403 catch exception 0404 disp(['Error getting dockserver files from ' dockserver.host ':']); 0405 disp(getReport(exception, 'extended')); 0406 end 0407 end 0408 new_xbds = [new_xbds{:}]; 0409 new_logs = [new_logs{:}]; 0410 disp(['Binary data files downloaded: ' num2str(numel(new_xbds)) '.']); 0411 disp(['Surface log files downloaded: ' num2str(numel(new_logs)) '.']); 0412 case {'seaglider'} 0413 new_engs = cell(size(config.basestations)); 0414 new_logs = cell(size(config.basestations)); 0415 for basestation_idx = 1:numel(config.basestations) 0416 basestation = config.dockservers(basestation_idx); 0417 try 0418 [new_engs{basestation_idx}, new_logs{basestation_idx}] = ... 0419 getDockserverFiles(basestation, glider_serial, ascii_dir, ascii_dir, ... 0420 'eng', file_options.eng_name_pattern, ... 0421 'log', file_options.log_name_pattern, ... 0422 'start', download_start, ... 0423 'final', download_final); 0424 catch exception 0425 disp(['Error getting basestation files from ' basestation.host ':']); 0426 disp(getReport(exception, 'extended')); 0427 end 0428 end 0429 new_engs = [new_engs{:}]; 0430 new_logs = [new_logs{:}]; 0431 disp(['Engineering data files downloaded: ' num2str(numel(new_engs)) '.']); 0432 disp(['Dive log data files downloaded: ' num2str(numel(new_logs)) '.']); 0433 case {'seaexplorer'} 0434 warning('glider_toolbox:main_glider_data_processing_dt:NotImplemented', ... 0435 'Real time file retrieval not implemented for SeaExplorer') 0436 otherwise 0437 end 0438 0439 0440 %% Convert binary glider files to ascii human readable format. 0441 % For Seaglider, do nothing but join the lists of new eng and log files. 0442 % For Slocum, convert each downloaded binary file to ascii format in the 0443 % ascii directory and store the returned absolute path for later use. 0444 % Since some conversion may fail use a cell array of string cell arrays and 0445 % flatten it when finished, leaving only the succesfully created dbas. 0446 % Give a second try to failing files, because they might have failed due to 0447 % a missing cache file generated later. 0448 switch glider_type 0449 case {'slocum_g1' 'slocum_g2'} 0450 disp('Converting binary data files to ascii format...'); 0451 new_files = cell(size(new_xbds)); 0452 for conversion_retry = 1:2 0453 for xbd_idx = 1:numel(new_xbds) 0454 if isempty(new_files{xbd_idx}) 0455 xbd_fullfile = new_xbds{xbd_idx}; 0456 [~, xbd_name, xbd_ext] = fileparts(xbd_fullfile); 0457 xbd_name_ext = [xbd_name xbd_ext]; 0458 dba_name_ext = regexprep(xbd_name_ext, ... 0459 file_options.xbd_name_pattern, ... 0460 file_options.dba_name_replace); 0461 dba_fullfile = fullfile(ascii_dir, dba_name_ext); 0462 try 0463 new_files{xbd_idx} = ... 0464 {xbd2dba(xbd_fullfile, dba_fullfile, 'cache', cache_dir, ... 0465 'cmdname', config.wrcprogs.dbd2asc)}; 0466 catch exception 0467 new_files{xbd_idx} = {}; 0468 if conversion_retry == 2 0469 disp(['Error converting binary file ' xbd_name_ext ':']); 0470 disp(getReport(exception, 'extended')); 0471 end 0472 end 0473 end 0474 end 0475 end 0476 new_files = [new_files{:}]; 0477 disp(['Binary files converted: ' ... 0478 num2str(numel(new_files)) ' of ' num2str(numel(new_xbds)) '.']); 0479 case {'seaglider'} 0480 new_files = [new_engs{:} new_logs{:}]; 0481 case {'seaexplorer'} 0482 warning('glider_toolbox:main_glider_data_processing_dt:SeaExplorerFilesRT', ... 0483 'Faking newly retrieved files with contents of ascii directory'); 0484 new_files = dir(ascii_dir); 0485 new_files = {new_files(~[new_files.isdir]).name}; 0486 otherwise 0487 end 0488 0489 0490 %% Load data from ascii deployment glider files if there is new data. 0491 if isempty(new_files) 0492 disp('No new deployment data, processing and product generation will be skipped.'); 0493 else 0494 disp('Loading raw deployment data from text files...'); 0495 load_start = utc2posixtime(deployment_start); 0496 load_final = posixtime(); 0497 if ~isnan(deployment_end) 0498 load_final = utc2posixtime(deployment_end); 0499 end 0500 try 0501 switch glider_type 0502 case {'slocum_g1' 'slocum_g2'} 0503 [meta_raw, data_raw] = ... 0504 loadSlocumData(ascii_dir, ... 0505 file_options.dba_name_pattern_nav, ... 0506 file_options.dba_name_pattern_sci, ... 0507 'timenav', file_options.dba_time_sensor_nav, ... 0508 'timesci', file_options.dba_time_sensor_sci, ... 0509 'sensors', file_options.dba_sensors, ... 0510 'period', [load_start load_final], ... 0511 'format', 'struct'); 0512 source_files = {meta_raw.headers.filename_label}; 0513 case 'seaglider' 0514 [meta_raw, data_raw] = ... 0515 loadSeagliderData(ascii_dir, ... 0516 file_options.log_name_pattern, ... 0517 file_options.eng_name_pattern, ... 0518 'columns', file_options.eng_columns, ... 0519 'params' , file_options.log_params, ... 0520 'period', [load_start load_final], ... 0521 'format', 'merged'); 0522 source_files = meta_raw.sources; 0523 case {'seaexplorer'} 0524 [meta_raw, data_raw] = ... 0525 loadSeaExplorerData(ascii_dir, ... 0526 file_options.gli_name_pattern, ... 0527 file_options.pld_name_pattern, ... 0528 'timegli', file_options.gli_time, ... 0529 'timepld', file_options.pld_time, ... 0530 'format', 'struct'); 0531 source_files = meta_raw.sources; 0532 otherwise 0533 warning('glider_toolbox:main_glider_data_processing_dt:InvalidGliderType', ... 0534 'Unknown glider model: %s.', glider_model); 0535 end 0536 catch exception 0537 disp('Error loading raw data:'); 0538 disp(getReport(exception, 'extended')); 0539 end 0540 end 0541 0542 0543 %% Add source files to deployment structure if loading succeeded. 0544 if ~isempty(source_files) 0545 deployment.source_files = sprintf('%s\n', source_files{:}); 0546 end 0547 0548 0549 %% Generate L0 NetCDF file (raw/preprocessed data), if needed and possible. 0550 if ~isempty(fieldnames(data_raw)) && ~isempty(netcdf_l0_file) 0551 disp('Generating NetCDF L0 output...'); 0552 try 0553 switch glider_type 0554 case {'slocum_g1' 'slocum_g2'} 0555 outputs.netcdf_l0 = generateOutputNetCDF( ... 0556 netcdf_l0_file, data_raw, meta_raw, deployment, ... 0557 netcdf_l0_options.variables, ... 0558 netcdf_l0_options.dimensions, ... 0559 netcdf_l0_options.attributes, ... 0560 'time', {'m_present_time' 'sci_m_present_time'}, ... 0561 'position', {'m_gps_lon' 'm_gps_lat'; 'm_lon' 'm_lat'}, ... 0562 'position_conversion', @nmea2deg, ... 0563 'vertical', {'m_depth' 'sci_water_pressure'}, ... 0564 'vertical_conversion', {[] @(z)(z * 10)}, ... 0565 'vertical_positive', {'down'} ); 0566 case 'seaglider' 0567 outputs.netcdf_l0 = generateOutputNetCDF( ... 0568 netcdf_l0_file, data_raw, meta_raw, deployment, ... 0569 netcdf_l0_options.variables, ... 0570 netcdf_l0_options.dimensions, ... 0571 netcdf_l0_options.attributes, ... 0572 'time', {'elaps_t'}, ... 0573 'time_conversion', @(t)(t + meta_raw.start_secs), ... 0574 'position', {'GPSFIX_fixlon' 'GPSFIX_fixlat'}, ... 0575 'position_conversion', @nmea2deg, ... 0576 'vertical', {'depth'}, ... 0577 'vertical_conversion', {@(z)(z * 10)}, ... 0578 'vertical_positive', {'down'} ); 0579 case {'seaexplorer'} 0580 outputs.netcdf_l0 = generateOutputNetCDF( ... 0581 netcdf_l0_file, data_raw, meta_raw, deployment, ... 0582 netcdf_l0_options.variables, ... 0583 netcdf_l0_options.dimensions, ... 0584 netcdf_l0_options.attributes, ... 0585 'time', {'Timestamp' 'PLD_REALTIMECLOCK'}, ... 0586 'position', {'NAV_LONGITUDE' 'NAV_LATITUDE'; 'Lon' 'Lat'}, ... 0587 'position_conversion', @nmea2deg, ... 0588 'vertical', {'Depth' 'SBD_PRESSURE'}, ... 0589 'vertical_conversion', {[] @(z)(z * 10)}, ... 0590 'vertical_positive', {'down'} ); 0591 end 0592 disp(['Output NetCDF L0 (raw data) generated: ' outputs.netcdf_l0 '.']); 0593 catch exception 0594 disp(['Error generating NetCDF L0 (raw data) output ' netcdf_l0_file ':']); 0595 disp(getReport(exception, 'extended')); 0596 end 0597 end 0598 0599 0600 %% Preprocess raw glider data. 0601 if ~isempty(fieldnames(data_raw)) 0602 disp('Preprocessing raw data...'); 0603 try 0604 switch glider_type 0605 case 'seaglider' 0606 seaglider_time_sensor_select = ... 0607 strcmp('elaps_t', {preprocessing_options.time_list.time}); 0608 preprocessing_options.time_list(seaglider_time_sensor_select).conversion = ... 0609 @(t)(t + meta_raw.start_secs); 0610 end 0611 [data_preprocessed, meta_preprocessed] = ... 0612 preprocessGliderData(data_raw, meta_raw, preprocessing_options); 0613 catch exception 0614 disp('Error preprocessing glider deployment data:'); 0615 disp(getReport(exception, 'extended')); 0616 end 0617 end 0618 0619 0620 %% Process preprocessed glider data. 0621 if ~isempty(fieldnames(data_preprocessed)) 0622 disp('Processing glider data...'); 0623 try 0624 [data_processed, meta_processed] = ... 0625 processGliderData(data_preprocessed, meta_preprocessed, processing_options); 0626 catch exception 0627 disp('Error processing glider deployment data:'); 0628 disp(getReport(exception, 'extended')); 0629 end 0630 end 0631 0632 0633 %% Generate L1 NetCDF file (processed data), if needed and possible. 0634 if ~isempty(fieldnames(data_processed)) && ~isempty(netcdf_l1_file) 0635 disp('Generating NetCDF L1 output...'); 0636 try 0637 outputs.netcdf_l1 = generateOutputNetCDF( ... 0638 netcdf_l1_file, data_processed, meta_processed, deployment, ... 0639 netcdf_l1_options.variables, ... 0640 netcdf_l1_options.dimensions, ... 0641 netcdf_l1_options.attributes); 0642 disp(['Output NetCDF L1 (processed data) generated: ' ... 0643 outputs.netcdf_l1 '.']); 0644 catch exception 0645 disp(['Error generating NetCDF L1 (processed data) output ' ... 0646 netcdf_l1_file ':']); 0647 disp(getReport(exception, 'extended')); 0648 end 0649 end 0650 0651 0652 %% Generate processed data figures. 0653 if ~isempty(fieldnames(data_processed)) && ~isempty(figure_dir) 0654 disp('Generating figures from processed data...'); 0655 try 0656 figures.figproc = generateGliderFigures( ... 0657 data_processed, figproc_options, ... 0658 'date', datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00'), ... 0659 'dirname', figure_dir); 0660 catch exception 0661 disp('Error generating processed data figures:'); 0662 disp(getReport(exception, 'extended')); 0663 end 0664 end 0665 0666 0667 %% Grid processed glider data. 0668 if ~isempty(fieldnames(data_processed)) 0669 disp('Gridding glider data...'); 0670 try 0671 [data_gridded, meta_gridded] = ... 0672 gridGliderData(data_processed, meta_processed, gridding_options); 0673 catch exception 0674 disp('Error gridding glider deployment data:'); 0675 disp(getReport(exception, 'extended')); 0676 end 0677 end 0678 0679 0680 %% Generate L2 (gridded data) netcdf file, if needed and possible. 0681 if ~isempty(fieldnames(data_gridded)) && ~isempty(netcdf_l2_file) 0682 disp('Generating NetCDF L2 output...'); 0683 try 0684 outputs.netcdf_l2 = generateOutputNetCDF( ... 0685 netcdf_l2_file, data_gridded, meta_gridded, deployment, ... 0686 netcdf_l2_options.variables, ... 0687 netcdf_l2_options.dimensions, ... 0688 netcdf_l2_options.attributes); 0689 disp(['Output NetCDF L2 (gridded data) generated: ' ... 0690 outputs.netcdf_l2 '.']); 0691 catch exception 0692 disp(['Error generating NetCDF L2 (gridded data) output ' ... 0693 netcdf_l2_file ':']); 0694 disp(getReport(exception, 'extended')); 0695 end 0696 end 0697 0698 0699 %% Generate gridded data figures. 0700 if ~isempty(fieldnames(data_gridded)) && ~isempty(figure_dir) 0701 disp('Generating figures from gridded data...'); 0702 try 0703 figures.figgrid = generateGliderFigures( ... 0704 data_gridded, figgrid_options, ... 0705 'date', datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00'), ... 0706 'dirname', figure_dir); 0707 catch exception 0708 disp('Error generating gridded data figures:'); 0709 disp(getReport(exception, 'extended')); 0710 end 0711 end 0712 0713 0714 %% Copy selected products to corresponding public location, if needed. 0715 if ~isempty(fieldnames(outputs)) 0716 disp('Copying public outputs...'); 0717 output_name_list = fieldnames(outputs); 0718 for output_name_idx = 1:numel(output_name_list) 0719 output_name = output_name_list{output_name_idx}; 0720 if isfield(config.paths_public, output_name) ... 0721 && ~isempty(config.paths_public.(output_name)) 0722 output_local_file = outputs.(output_name); 0723 output_public_file = ... 0724 strfstruct(config.paths_public.(output_name), deployment); 0725 output_public_dir = fileparts(output_public_file); 0726 [status, attrout] = fileattrib(output_public_dir); 0727 if ~status 0728 [status, message] = mkdir(output_public_dir); 0729 elseif ~attrout.directory 0730 status = false; 0731 message = 'not a directory'; 0732 end 0733 if status 0734 [success, message] = copyfile(output_local_file, output_public_file); 0735 if success 0736 disp(['Public output ' output_name ' succesfully copied: ' ... 0737 output_public_file '.']); 0738 else 0739 disp(['Error creating public copy of deployment product ' ... 0740 output_name ': ' output_public_file '.']); 0741 disp(message); 0742 end 0743 else 0744 disp(['Error creating public output directory ' ... 0745 output_public_dir ':']); 0746 disp(message); 0747 end 0748 end 0749 end 0750 end 0751 0752 0753 %% Copy selected figures to its public location, if needed. 0754 % Copy all generated figures or only the ones in the include list (if any) 0755 % excluding the ones in the exclude list. 0756 if ~isempty(fieldnames(figures)) ... 0757 && isfield(config.paths_public, 'figure_dir') ... 0758 && ~isempty(config.paths_public.figure_dir) 0759 disp('Copying public figures...'); 0760 public_figure_baseurl = ... 0761 strfstruct(config.paths_public.figure_url, deployment); 0762 public_figure_dir = ... 0763 strfstruct(config.paths_public.figure_dir, deployment); 0764 public_figure_include_all = true; 0765 public_figure_exclude_none = true; 0766 public_figure_include_list = []; 0767 public_figure_exclude_list = []; 0768 if isfield(config.paths_public, 'figure_include') 0769 public_figure_include_all = false; 0770 public_figure_include_list = config.paths_public.figure_include; 0771 end 0772 if isfield(config.paths_public, 'figure_exclude') 0773 public_figure_exclude_none = false; 0774 public_figure_exclude_list = config.paths_public.figure_exclude; 0775 end 0776 public_figures = struct(); 0777 public_figures_local = struct(); 0778 figure_output_name_list = fieldnames(figures); 0779 for figure_output_name_idx = 1:numel(figure_output_name_list) 0780 figure_output_name = figure_output_name_list{figure_output_name_idx}; 0781 figure_output = figures.(figure_output_name); 0782 figure_name_list = fieldnames(figure_output); 0783 for figure_name_idx = 1:numel(figure_name_list) 0784 figure_name = figure_name_list{figure_name_idx}; 0785 if (public_figure_include_all ... 0786 || ismember(figure_name, public_figure_include_list)) ... 0787 && (public_figure_exclude_none ... 0788 || ~ismember(figure_name, public_figure_exclude_list)) 0789 if isfield(public_figures_local, figure_name) 0790 disp(['Warning: figure ' figure_name ' appears to be duplicated.']); 0791 else 0792 public_figures_local.(figure_name) = figure_output.(figure_name); 0793 end 0794 end 0795 end 0796 end 0797 public_figure_name_list = fieldnames(public_figures_local); 0798 if ~isempty(public_figure_name_list) 0799 [status, attrout] = fileattrib(public_figure_dir); 0800 if ~status 0801 [status, message] = mkdir(public_figure_dir); 0802 elseif ~attrout.directory 0803 status = false; 0804 message = 'not a directory'; 0805 end 0806 if status 0807 for public_figure_name_idx = 1:numel(public_figure_name_list) 0808 public_figure_name = public_figure_name_list{public_figure_name_idx}; 0809 figure_local = public_figures_local.(public_figure_name); 0810 figure_public = figure_local; 0811 figure_public.url = ... 0812 [public_figure_baseurl '/' ... 0813 figure_public.filename '.' figure_public.format]; 0814 figure_public.dirname = public_figure_dir; 0815 figure_public.fullfile = ... 0816 fullfile(figure_public.dirname, ... 0817 [figure_public.filename '.' figure_public.format]); 0818 [success, message] = ... 0819 copyfile(figure_local.fullfile, figure_public.fullfile); 0820 if success 0821 public_figures.(public_figure_name) = figure_public; 0822 disp(['Public figure ' public_figure_name ' succesfully copied.']); 0823 else 0824 disp(['Error creating public copy of figure ' ... 0825 public_figure_name ': ' figure_public.fullfile '.']); 0826 disp(message); 0827 end 0828 end 0829 else 0830 disp(['Error creating public figure directory ' public_figure_dir ':']); 0831 disp(message); 0832 end 0833 end 0834 % Write the figure information to the JSON service file. 0835 if isfield(config.paths_public, 'figure_info') ... 0836 && ~isempty(config.paths_public.figure_info) 0837 disp('Generating figure information service file...'); 0838 public_figure_info_file = ... 0839 strfstruct(config.paths_public.figure_info, deployment); 0840 try 0841 savejson(public_figures, public_figure_info_file); 0842 disp(['Figure information service file successfully generated: ' ... 0843 public_figure_info_file]); 0844 catch exception 0845 disp(['Error creating figure information service file ' ... 0846 public_figure_info_file ':']); 0847 disp(message); 0848 end 0849 end 0850 end 0851 0852 0853 %% Stop deployment processing logging. 0854 disp(['Deployment processing end time: ' ... 0855 datestr(posixtime2utc(posixtime()), 'yyyy-mm-ddTHH:MM:SS+00:00')]); 0856 diary('off'); 0857 0858 end