getBasestationFiles

GETBASESTATIONFILES  Get Seaglider eng and log data files from basestation through (S)FTP.

  Syntax:
    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR)
    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPTIONS)
    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPT1, VAL1, ...)

  Description:
    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR)
    retrieves Seaglider dive data files (.eng) and dive log files (.log) from
    the glider with serial code GLIDER from the basestation defined by struct 
    BASESTATION to local directories ENG_DIR and LOG_DIR respectively, and
    returns the list of downloaded files in string cell arrays ENGS and LOGS.
    Existing files in the local directories are updated only if they are 
    smaller than remote ones.

  BASESTATION is a struct with the fields needed by functions FTP or SFTP:
    HOST: url as either fully qualified name or IP with optional port (string).
    USER: user to access the basestation if needed (string).
    PASS: password of the basestation if needed (string).
    CONN: name or handle of connection type function, @FTP (default) or @SFTP.

    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPTIONS) and 
    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPT1, VAL1, ...)
    accept the following options, given in key-value pairs OPT1, VAL1...
    or in a struct OPTIONS with field names as option keys and field values
    as option values, allowing to restrict the set of files to download:
      ENG: engineering data file name pattern.
        Download engineering files matching given pattern only.
        Its value may be any valid regular expression string or empty.
        If empty, no engineering data files are downloaded.
        Default value: ['^p' GLIDER '\d+\.eng$']
      LOG: log file name pattern.
        Download log data files matching given pattern only.
        Its value may be any valid regular expression string.
        If empty, no log files are downloaded.
        Default value: ['^p' GLIDER '\d+\.log$']
      START: initial date of the period of interest.
        If given, do not download files before the given date.
        It may be any valid input compatible with ENG2DATE and LOG2DATE
        options below, usually a serial date number.
        Default value: -Inf
      FINAL: final date of the period of interest.
        If given, do not download files after the the given date.
        It may be any valid input compatible with ENG2DATE and LOG2DATE
        options below, usually a serial date number.
        Default value: +Inf
      ENG2DATE: date of binary file.
        If date filtering is enabled, use the given function
        to extract the date of an eng data file from its attributes.
        The function receives a struct in the format returned by function DIR
        and should return a date in a format comparable to START and FINAL.
        Default value: modification time (see note on date filtering)
      LOG2DATE: date of log file.
        If date filtering is enabled, use the given function
        to extract the date of a log file from its attribtues.
        The function receives a struct in the format returned by function DIR
        and should return a date in a format comparable to START and FINAL.
        Default value: modification time (see note on date filtering)

  Notes:
    By default, date filtering is done based on the modification time
    of the remote files. Filtering by dive number is also possible,
    by providing a function to extract the dive number from the file name 
    in options ENG2DATE and LOG2DATE and define the dive range in options 
    START and FINAL.

  Examples:
    basestation.host = 'ftp.mybasestation.org'
    basestation.user = 'myself'
    basestation.pass = 'top_secret'   
    glider = 'happyglider'
    eng_dir = 'funnymission/ascii'
    log_dir = 'funnymission/ascii'
    % Get all engineering data files and log data files.
    [engs, logs] = getBasestationFiles(basestation, glider, eng_dir, log_dir)
    % Get only eng files and no log files for even dives during last week:
    [engs, logs] = ...
      getBasestationFiles(basestation, glider, eng_dir, log_dir, ...
                          'eng', '^p\d+[02468].eng$', 'log', [], ...
                          'start', now()-7, 'final', now())

  See also:
    FTP
    SFTP
    DIR

  Authors:
    Joan Pau Beltran  <joanpau.beltran@socib.cat>

0001 function [engs, logs] = getBasestationFiles(basestation, glider, eng_dir, log_dir, varargin)
0002 %GETBASESTATIONFILES  Get Seaglider eng and log data files from basestation through (S)FTP.
0003 %
0004 %  Syntax:
0005 %    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR)
0006 %    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPTIONS)
0007 %    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPT1, VAL1, ...)
0008 %
0009 %  Description:
0010 %    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR)
0011 %    retrieves Seaglider dive data files (.eng) and dive log files (.log) from
0012 %    the glider with serial code GLIDER from the basestation defined by struct
0013 %    BASESTATION to local directories ENG_DIR and LOG_DIR respectively, and
0014 %    returns the list of downloaded files in string cell arrays ENGS and LOGS.
0015 %    Existing files in the local directories are updated only if they are
0016 %    smaller than remote ones.
0017 %
0018 %  BASESTATION is a struct with the fields needed by functions FTP or SFTP:
0019 %    HOST: url as either fully qualified name or IP with optional port (string).
0020 %    USER: user to access the basestation if needed (string).
0021 %    PASS: password of the basestation if needed (string).
0022 %    CONN: name or handle of connection type function, @FTP (default) or @SFTP.
0023 %
0024 %    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPTIONS) and
0025 %    [ENGS, LOGS] = GETBASESTATIONFILES(BASESTATION, GLIDER, ENG_DIR, LOG_DIR, OPT1, VAL1, ...)
0026 %    accept the following options, given in key-value pairs OPT1, VAL1...
0027 %    or in a struct OPTIONS with field names as option keys and field values
0028 %    as option values, allowing to restrict the set of files to download:
0029 %      ENG: engineering data file name pattern.
0030 %        Download engineering files matching given pattern only.
0031 %        Its value may be any valid regular expression string or empty.
0032 %        If empty, no engineering data files are downloaded.
0033 %        Default value: ['^p' GLIDER '\d+\.eng$']
0034 %      LOG: log file name pattern.
0035 %        Download log data files matching given pattern only.
0036 %        Its value may be any valid regular expression string.
0037 %        If empty, no log files are downloaded.
0038 %        Default value: ['^p' GLIDER '\d+\.log$']
0039 %      START: initial date of the period of interest.
0040 %        If given, do not download files before the given date.
0041 %        It may be any valid input compatible with ENG2DATE and LOG2DATE
0042 %        options below, usually a serial date number.
0043 %        Default value: -Inf
0044 %      FINAL: final date of the period of interest.
0045 %        If given, do not download files after the the given date.
0046 %        It may be any valid input compatible with ENG2DATE and LOG2DATE
0047 %        options below, usually a serial date number.
0048 %        Default value: +Inf
0049 %      ENG2DATE: date of binary file.
0050 %        If date filtering is enabled, use the given function
0051 %        to extract the date of an eng data file from its attributes.
0052 %        The function receives a struct in the format returned by function DIR
0053 %        and should return a date in a format comparable to START and FINAL.
0054 %        Default value: modification time (see note on date filtering)
0055 %      LOG2DATE: date of log file.
0056 %        If date filtering is enabled, use the given function
0057 %        to extract the date of a log file from its attribtues.
0058 %        The function receives a struct in the format returned by function DIR
0059 %        and should return a date in a format comparable to START and FINAL.
0060 %        Default value: modification time (see note on date filtering)
0061 %
0062 %  Notes:
0063 %    By default, date filtering is done based on the modification time
0064 %    of the remote files. Filtering by dive number is also possible,
0065 %    by providing a function to extract the dive number from the file name
0066 %    in options ENG2DATE and LOG2DATE and define the dive range in options
0067 %    START and FINAL.
0068 %
0069 %  Examples:
0070 %    basestation.host = 'ftp.mybasestation.org'
0071 %    basestation.user = 'myself'
0072 %    basestation.pass = 'top_secret'
0073 %    glider = 'happyglider'
0074 %    eng_dir = 'funnymission/ascii'
0075 %    log_dir = 'funnymission/ascii'
0076 %    % Get all engineering data files and log data files.
0077 %    [engs, logs] = getBasestationFiles(basestation, glider, eng_dir, log_dir)
0078 %    % Get only eng files and no log files for even dives during last week:
0079 %    [engs, logs] = ...
0080 %      getBasestationFiles(basestation, glider, eng_dir, log_dir, ...
0081 %                          'eng', '^p\d+[02468].eng$', 'log', [], ...
0082 %                          'start', now()-7, 'final', now())
0083 %
0084 %  See also:
0085 %    FTP
0086 %    SFTP
0087 %    DIR
0088 %
0089 %  Authors:
0090 %    Joan Pau Beltran  <joanpau.beltran@socib.cat>
0091 
0092 %  Copyright (C) 2014-2016
0093 %  ICTS SOCIB - Servei d'observacio i prediccio costaner de les Illes Balears
0094 %  <http://www.socib.es>
0095 %
0096 %  This program is free software: you can redistribute it and/or modify
0097 %  it under the terms of the GNU General Public License as published by
0098 %  the Free Software Foundation, either version 3 of the License, or
0099 %  (at your option) any later version.
0100 %
0101 %  This program is distributed in the hope that it will be useful,
0102 %  but WITHOUT ANY WARRANTY; without even the implied warranty of
0103 %  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
0104 %  GNU General Public License for more details.
0105 %
0106 %  You should have received a copy of the GNU General Public License
0107 %  along with this program.  If not, see <http://www.gnu.org/licenses/>.
0108 
0109   error(nargchk(4, 12, nargin, 'struct'));
0110 
0111   
0112   %% Set options and default values.
0113   options.start = -Inf;
0114   options.final = +Inf;
0115   options.eng = ['^p' glider '\d+\.eng$'];
0116   options.log = ['^p' glider '\d+\.log$'];
0117   options.eng2date = @(f)(f.datenum);
0118   options.log2date = @(f)(f.datenum);
0119   
0120   
0121   %% Parse optional arguments.
0122   % Get option key-value pairs in any accepted call signature.
0123   argopts = varargin;
0124   if isscalar(argopts) && isstruct(argopts{1})
0125     % Options passed as a single option struct argument:
0126     % field names are option keys and field values are option values.
0127     opt_key_list = fieldnames(argopts{1});
0128     opt_val_list = struct2cell(argopts{1});
0129   elseif mod(numel(argopts), 2) == 0
0130     % Options passed as key-value argument pairs.
0131     opt_key_list = argopts(1:2:end);
0132     opt_val_list = argopts(2:2:end);
0133   else
0134     error('glider_toolbox:getBasestationFiles:InvalidOptions', ...
0135           'Invalid optional arguments (neither key-value pairs nor struct).');
0136   end
0137   % Overwrite default options with values given in extra arguments.
0138   for opt_idx = 1:numel(opt_key_list)
0139     opt = lower(opt_key_list{opt_idx});
0140     val = opt_val_list{opt_idx};
0141     if isfield(options, opt)
0142       options.(opt) = val;
0143     else
0144       error('glider_toolbox:getBasestationFiles:InvalidOption', ...
0145             'Invalid option: %s.', opt);
0146     end
0147   end
0148 
0149 
0150   %% Basestation (remote) directory definition.
0151   % Consider pass these paths as (maybe optional) input arguments.
0152   remote_base_dir = '/home';
0153   remote_eng_dir = [remote_base_dir '/' 'sg' glider];
0154   remote_log_dir = [remote_base_dir '/' 'sg' glider];
0155 
0156 
0157   %% Collect some parameters given in options.
0158   eng_name = options.eng;
0159   log_name = options.log;
0160   eng_newfunc = [];
0161   log_newfunc = [];
0162   updatefunc = @(l,r)(l.bytes < r.bytes);
0163   if isfinite(options.start) || isfinite(options.final)
0164     eng_newfunc = @(r)(options.start <= options.eng2date(r) && ...
0165                        options.xbd2date(r) <= options.final);
0166     log_newfunc = @(r)(options.start <= options.log2date(r) && ...
0167                        options.log2date(r) <= options.final);
0168   end
0169 
0170 
0171   %% Open (S)FTP connection.
0172   host = basestation.host;
0173   user = [];
0174   pass = [];
0175   conn = @ftp;
0176   if isfield(basestation, 'user') && ~isequal(basestation.user, [])
0177     user = basestation.user;
0178   end
0179   if isfield(basestation, 'pass') && ~isequal(basestation.pass, [])
0180     pass = basestation.pass;
0181   end
0182   if isfield(basestation, 'conn') && ~isequal(basestation.conn, [])
0183     conn = basestation.conn;
0184     if ischar(conn)
0185       conn = str2func(conn);
0186     end
0187   end
0188   disp(['Connecting to host ' host '...']);
0189   ftp_handle = conn(host, user, pass);
0190 
0191 
0192   %% Engineering data file download.
0193   disp('Downloading engineering data files...');
0194   engs = {};
0195   if ~isequal(eng_name, [])
0196     try
0197       engs = getfiles(ftp_handle, 'target', eng_dir, ...
0198                       'source', remote_eng_dir, 'include', eng_name, ...
0199                       'new', eng_newfunc, 'update', updatefunc);
0200     catch exception
0201       warning('glider_toolbox:getBasestationFiles:DownloadError', ...
0202               'Error downloading engineering data files: %s', exception.message);
0203     end
0204     disp([num2str(numel(engs)) ' new/updated engineering data files fetched.']);
0205   end
0206 
0207 
0208   %% Log data file download.
0209   disp('Downloading log data files...');
0210   logs = {};
0211   if ~isequal(log_name, [])
0212     try
0213       logs = getfiles(ftp_handle, 'target', log_dir, ...
0214                       'source', remote_log_dir, 'include', log_name, ...
0215                       'new', log_newfunc, 'update', updatefunc);
0216     catch exception
0217       warning('glider_toolbox:getBasestationFiles:DownloadError', ...
0218               'Error downloading log data files: %s.', exception.message);
0219     end
0220     disp([num2str(numel(logs)) ' new/updated log data files fetched.']);
0221   end
0222 
0223   
0224   %% Close ftp connection.
0225   close(ftp_handle);
0226   disp(['Closed connection to host ' basestation.host '.']);
0227 
0228 end