getDockserverFiles

GETDOCKSERVERFILES  Get binary data files and surface log files from dockserver through (S)FTP.

  Syntax:
    [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR)
    [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPTIONS)
    [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPT1, VAL1, ...)

  [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR) 
  retrieves new binary files (.[smdtne]bd) and surface dialog files from the
  glider named GLIDER from the remote dockserver defined by struct DOCKSERVER
  to local directories XBD_DIR and LOG_DIR respectively, and returns the list
  of downloaded files in string cell arrays XBDS and LOGS. Existing files in 
  the local directories are updated only if they are smaller than remote ones.

  DOCKSERVER is a struct with the fields needed by functions FTP or SFTP:
    HOST: url as either fully qualified name or IP with optional port (string).
    USER: user to access the dockserver if needed (string).
    PASS: password of the dockserver if needed (string).
    CONN: name or handle of connection type function, @FTP (default) or @SFTP.

  [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPTIONS) and
  [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPT1, VAL1, ...)
  accept the following options, given in key-value pairs OPT1, VAL1... or in a
  struct OPTIONS with field names as option keys and field values as option 
  values, allowing to restrict the set of files to download:
    XBD: binary file name pattern.
      Download binary files matching given pattern only.
      Its value may be any valid regular expression string or empty.
      If empty no binary files are downloaded.
      Default value: '^.+\.[smdtne]bd$'
    LOG: log file name pattern.
      Download log files matching given pattern only.
      Its value may be any valid regular expression string or empty.
      If empty no log files are downloaded.
      Default value: '^.+\.log$' 
    START: initial date of the period of interest.
      If given, do not download files before the given date.
      It may be any valid input compatible with XBD2DATE and LOG2DATE
      options below, usually a serial date number.
      Default value: -Inf
    FINAL: final date of the period of interest.
      If given, do not download files after the the given date.
      It may be any valid input compatible with XBD2DATE and LOG2DATE
      options below, usually a serial date number.
      Default value: +Inf
    XBD2DATE: date of binary file.
      If date filtering is enabled, use the given function
      to extract the date of a binary file from its attributes.
      The function receives a struct in the format returned by function DIR
      and should return a date in a format comparable to START and FINAL.
      Default value: date from file name (see note on date filtering)
    LOG2DATE: date of log file.
      If date filtering is enabled, use the given function
      to extract the date of a log file from its attribtues.
      The function receives a struct in the format returned by function DIR
      and should return a date in a format comparable to START and FINAL.
      Default value: date from file name (see note on date filtering)

  Notes:
    By default, date filtering is done based on the mission date computed
    from the file names, not on the modification time. It relies on remote
    file names having the conventional Slocum file name format.
    For binary files it is:
      ru07-2011-347-4-0.sbd
    where
      ru07: glider name.
      2011: year in which the mission was started.
      347: zero-based day of the year on which the mission was started.
      4: zero-based mission number for the day the mission was started.
      0: zero-based segment number of the current mission number.
    For log files it is:
      icoast00_modem_20120510T091438.log
    where
      icoast00: glider name.
      modem: transmission method ('modem' or 'network').
      20120510T091438: ISO 8601 UTC timestamp.

    This function is based on the previous work by Tomeu Garau. He is the true
    glider man.

  Examples:
    dockserver.host = 'ftp.mydockserver.org'
    dockserver.user = 'myself'
    dockserver.pass = 'top_secret'   
    glider = 'happyglider'
    xbd_dir = 'funnymission/binary'
    log_dir = 'funnymission/log'
    % Get all binary and log files.
    [xbds, logs] = getDockserverFiles(dockserver, glider, xbd_dir, log_dir)
    % Get only small files and no logs from missions started last week:
    [xbds, logs] = ...
      getDockserverFiles(dockserver, glider, xbd_dir, log_dir, ...
                         'xbd', '^*.[st]bd$', 'log', [], ...
                         'start', now()-7, 'final', now())

  See also:
    FTP
    SFTP
    DIR
    REGEX

  Authors:
    Joan Pau Beltran  <joanpau.beltran@socib.cat>

0001 function [xbds, logs] = getDockserverFiles(dockserver, glider, xbd_dir, log_dir, varargin)
0002 %GETDOCKSERVERFILES  Get binary data files and surface log files from dockserver through (S)FTP.
0003 %
0004 %  Syntax:
0005 %    [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR)
0006 %    [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPTIONS)
0007 %    [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPT1, VAL1, ...)
0008 %
0009 %  [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR)
0010 %  retrieves new binary files (.[smdtne]bd) and surface dialog files from the
0011 %  glider named GLIDER from the remote dockserver defined by struct DOCKSERVER
0012 %  to local directories XBD_DIR and LOG_DIR respectively, and returns the list
0013 %  of downloaded files in string cell arrays XBDS and LOGS. Existing files in
0014 %  the local directories are updated only if they are smaller than remote ones.
0015 %
0016 %  DOCKSERVER is a struct with the fields needed by functions FTP or SFTP:
0017 %    HOST: url as either fully qualified name or IP with optional port (string).
0018 %    USER: user to access the dockserver if needed (string).
0019 %    PASS: password of the dockserver if needed (string).
0020 %    CONN: name or handle of connection type function, @FTP (default) or @SFTP.
0021 %
0022 %  [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPTIONS) and
0023 %  [XBDS, LOGS] = GETDOCKSERVERFILES(DOCKSERVER, GLIDER, XBD_DIR, LOG_DIR, OPT1, VAL1, ...)
0024 %  accept the following options, given in key-value pairs OPT1, VAL1... or in a
0025 %  struct OPTIONS with field names as option keys and field values as option
0026 %  values, allowing to restrict the set of files to download:
0027 %    XBD: binary file name pattern.
0028 %      Download binary files matching given pattern only.
0029 %      Its value may be any valid regular expression string or empty.
0030 %      If empty no binary files are downloaded.
0031 %      Default value: '^.+\.[smdtne]bd$'
0032 %    LOG: log file name pattern.
0033 %      Download log files matching given pattern only.
0034 %      Its value may be any valid regular expression string or empty.
0035 %      If empty no log files are downloaded.
0036 %      Default value: '^.+\.log$'
0037 %    START: initial date of the period of interest.
0038 %      If given, do not download files before the given date.
0039 %      It may be any valid input compatible with XBD2DATE and LOG2DATE
0040 %      options below, usually a serial date number.
0041 %      Default value: -Inf
0042 %    FINAL: final date of the period of interest.
0043 %      If given, do not download files after the the given date.
0044 %      It may be any valid input compatible with XBD2DATE and LOG2DATE
0045 %      options below, usually a serial date number.
0046 %      Default value: +Inf
0047 %    XBD2DATE: date of binary file.
0048 %      If date filtering is enabled, use the given function
0049 %      to extract the date of a binary file from its attributes.
0050 %      The function receives a struct in the format returned by function DIR
0051 %      and should return a date in a format comparable to START and FINAL.
0052 %      Default value: date from file name (see note on date filtering)
0053 %    LOG2DATE: date of log file.
0054 %      If date filtering is enabled, use the given function
0055 %      to extract the date of a log file from its attribtues.
0056 %      The function receives a struct in the format returned by function DIR
0057 %      and should return a date in a format comparable to START and FINAL.
0058 %      Default value: date from file name (see note on date filtering)
0059 %
0060 %  Notes:
0061 %    By default, date filtering is done based on the mission date computed
0062 %    from the file names, not on the modification time. It relies on remote
0063 %    file names having the conventional Slocum file name format.
0064 %    For binary files it is:
0065 %      ru07-2011-347-4-0.sbd
0066 %    where
0067 %      ru07: glider name.
0068 %      2011: year in which the mission was started.
0069 %      347: zero-based day of the year on which the mission was started.
0070 %      4: zero-based mission number for the day the mission was started.
0071 %      0: zero-based segment number of the current mission number.
0072 %    For log files it is:
0073 %      icoast00_modem_20120510T091438.log
0074 %    where
0075 %      icoast00: glider name.
0076 %      modem: transmission method ('modem' or 'network').
0077 %      20120510T091438: ISO 8601 UTC timestamp.
0078 %
0079 %    This function is based on the previous work by Tomeu Garau. He is the true
0080 %    glider man.
0081 %
0082 %  Examples:
0083 %    dockserver.host = 'ftp.mydockserver.org'
0084 %    dockserver.user = 'myself'
0085 %    dockserver.pass = 'top_secret'
0086 %    glider = 'happyglider'
0087 %    xbd_dir = 'funnymission/binary'
0088 %    log_dir = 'funnymission/log'
0089 %    % Get all binary and log files.
0090 %    [xbds, logs] = getDockserverFiles(dockserver, glider, xbd_dir, log_dir)
0091 %    % Get only small files and no logs from missions started last week:
0092 %    [xbds, logs] = ...
0093 %      getDockserverFiles(dockserver, glider, xbd_dir, log_dir, ...
0094 %                         'xbd', '^*.[st]bd$', 'log', [], ...
0095 %                         'start', now()-7, 'final', now())
0096 %
0097 %  See also:
0098 %    FTP
0099 %    SFTP
0100 %    DIR
0101 %    REGEX
0102 %
0103 %  Authors:
0104 %    Joan Pau Beltran  <joanpau.beltran@socib.cat>
0105 
0106 %  Copyright (C) 2013-2016
0107 %  ICTS SOCIB - Servei d'observacio i prediccio costaner de les Illes Balears
0108 %  <http://www.socib.es>
0109 %
0110 %  This program is free software: you can redistribute it and/or modify
0111 %  it under the terms of the GNU General Public License as published by
0112 %  the Free Software Foundation, either version 3 of the License, or
0113 %  (at your option) any later version.
0114 %
0115 %  This program is distributed in the hope that it will be useful,
0116 %  but WITHOUT ANY WARRANTY; without even the implied warranty of
0117 %  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
0118 %  GNU General Public License for more details.
0119 %
0120 %  You should have received a copy of the GNU General Public License
0121 %  along with this program.  If not, see <http://www.gnu.org/licenses/>.
0122 
0123   error(nargchk(4, 16, nargin, 'struct'));
0124 
0125   
0126   %% Set options and default values.
0127   options.start = -Inf;
0128   options.final = +Inf;
0129   options.xbd = '^.+\.[smdtne]bd$';
0130   options.log = '^.+\.log$';
0131   options.xbd2date = ...
0132     @(f)(datenum(str2double(regexp(f.name, '^.*-(\d{4})-(\d{3})-\d+-\d+\.[smdtne]bd$', ...
0133                                    'tokens','once')) * [1 0 0; 0 0 1] + [0 0 1]));
0134   options.log2date = ...
0135     @(f)(datenum(str2double(regexp(f.name, '^.*_.*_(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})\.log$', ...
0136                                    'tokens','once'))));
0137   
0138   
0139   %% Parse optional arguments.
0140   % Get option key-value pairs in any accepted call signature.
0141   argopts = varargin;
0142   if isscalar(argopts) && isstruct(argopts{1})
0143     % Options passed as a single option struct argument:
0144     % field names are option keys and field values are option values.
0145     opt_key_list = fieldnames(argopts{1});
0146     opt_val_list = struct2cell(argopts{1});
0147   elseif mod(numel(argopts), 2) == 0
0148     % Options passed as key-value argument pairs.
0149     opt_key_list = argopts(1:2:end);
0150     opt_val_list = argopts(2:2:end);
0151   else
0152     error('glider_toolbox:getDockserverFiles:InvalidOptions', ...
0153           'Invalid optional arguments (neither key-value pairs nor struct).');
0154   end
0155   % Overwrite default options with values given in extra arguments.
0156   for opt_idx = 1:numel(opt_key_list)
0157     opt = lower(opt_key_list{opt_idx});
0158     val = opt_val_list{opt_idx};
0159     if isfield(options, opt)
0160       options.(opt) = val;
0161     else
0162       error('glider_toolbox:getDockserverFiles:InvalidOption', ...
0163             'Invalid option: %s.', opt);
0164     end
0165   end
0166 
0167 
0168   %% Dockserver (remote) directory definition.
0169   % Consider pass these paths as (maybe optional) input arguments.
0170   % Old dockservers used this other base path:
0171   %remote_base_dir = '/home/dockserver/gliders';
0172   remote_base_dir = '/var/opt/gmc/gliders';
0173   remote_xbd_dir = [remote_base_dir '/' lower(glider) '/' 'from-glider'];
0174   remote_log_dir = [remote_base_dir '/' lower(glider) '/' 'logs'];
0175 
0176 
0177   %% Collect some parameters given in options.
0178   xbd_name = options.xbd;
0179   log_name = options.log;
0180   xbd_newfunc = [];
0181   log_newfunc = [];
0182   updatefunc = @(l,r)(l.bytes < r.bytes);
0183   if isfinite(options.start) || isfinite(options.final)
0184     xbd_newfunc = @(r)(options.start <= options.xbd2date(r) && ...
0185                        options.xbd2date(r) <= options.final);
0186     log_newfunc = @(r)(options.start <= options.log2date(r) && ...
0187                        options.log2date(r) <= options.final);
0188   end
0189 
0190 
0191   %% Open (S)FTP connection.
0192   host = dockserver.host;
0193   user = [];
0194   pass = [];
0195   conn = @ftp;
0196   if isfield(dockserver, 'user') && ~isequal(dockserver.user, [])
0197     user = dockserver.user;
0198   end
0199   if isfield(dockserver, 'pass') && ~isequal(dockserver.pass, [])
0200     pass = dockserver.pass;
0201   end
0202   if isfield(dockserver, 'conn') && ~isequal(dockserver.conn, [])
0203     conn = dockserver.conn;
0204     if ischar(conn)
0205       conn = str2func(conn);
0206     end
0207   end
0208   disp(['Connecting to host ' host '...']);
0209   ftp_handle = conn(host, user, pass);
0210 
0211 
0212   %% Binary data file download.
0213   disp('Downloading binary data files...');
0214   xbds = {};
0215   if ~isequal(xbd_name, [])
0216     try
0217      xbds = getfiles(ftp_handle, 'target', xbd_dir, ...
0218                      'source', remote_xbd_dir, 'include', xbd_name, ...
0219                      'new', xbd_newfunc, 'update', updatefunc);
0220     catch exception
0221       warning('glider_toolbox:getDockserverFiles:DownloadError', ...
0222               'Error downloading binary data files: %s', exception.message);
0223     end
0224     disp([num2str(numel(xbds)) ' new/updated binary data files fetched.']);
0225   end
0226 
0227 
0228   %% Surface log file download.
0229   disp('Downloading surface log files...');
0230   logs = {};
0231   if ~isequal(xbd_name, [])
0232     try
0233      logs = getfiles(ftp_handle, 'target', log_dir, ...
0234                      'source', remote_log_dir, 'include', log_name, ...
0235                      'new', log_newfunc, 'update', updatefunc);
0236     catch exception
0237       warning('glider_toolbox:getDockserverFiles:DownloadError', ...
0238               'Error downloading surface log files: %s.', exception.message);
0239     end
0240     disp([num2str(numel(logs)) ' new/updated surface log files fetched.']);
0241   end
0242 
0243   
0244   %% Close ftp connection.
0245   close(ftp_handle);
0246   disp(['Closed connection to host ' dockserver.host '.']);
0247 
0248 end
0249