In Situ Thematic Assembly Centre for Marine Service

In Situ TAC

What is
In Situ TAC ?

In Situ TAC is the IN SITU OBSERVATIONAL COMPONENT of The Copernicus Marine Environment Monitoring Service.

Who is
In Situ TAC ?

What is
In Situ TAC doing?

In Situ TAC integrates thousand of local data from on-site sensors on board of a wide range of platforms

What have
In Situ TAC done so far?

Since 1950 ...

Where is
In Situ TAC
data ?

A regional approach ...

Distribution Units are INSTAC responsibles for providing data on a defined geographical area.

  • Global
  • + 6 regions:
  • Arctic
  • Baltic
  • North West Shelves
  • Iberia-Biscay-Ireland
  • Mediterranean
  • Black Sea

For each region, the In Situ Distribution Unit distributes all its data and metadata products on a FTP server (regional portal):

Important! FTPs are secured. First register as CMEMS user to get the access granted.

Data products available on those FTPs keep exactly the same:

  • data format (netCDF - OceanSites 1.2)
  • data flagging convention
  • ftp structure
  • file naming convention

in order to facilitate selection and processing to final users.

Same data format

For each region, the In Situ Distribution Unit distributes all its data and metadata products in netCDFs, format used by up to 1300 institutions for creating, accessing and sharing array-oriented scientific data.

The data files format is an implementation of NetCDF OceanSITES format.

CMEMS list a number of tools users can use for visualization & manipulation of such files

Our contribution for spreading the word about how-to-handle netCDF format can be found in the material build for training sessions in form of :

flagging convention

NRT quality controls are available here

Quality control flag scale:
Code Meaning Comment
0 No QC performed -
1 Good data All real-time QC tests passed
2 Probably good data -
3 Bad data that are potentially correctable These data are not to be used without scientific correction

flagging convention

Code Meaning Comment
4 Bad data Data have failed one or more of the tests
5 Value changed Data may be recovered after transmission error
6 Not used -
7 Nominal value Data were not observed but reported.Example: an instrument target depth
8 Interpolated value Missing data may be interpolated from neighbouring data in space or time
9 Missing value The value is missing, is not reported, is not applicable

same ftp structure

Each DU distributes all its data and metadata products in NetCDF files on an FTP server sticking with the same structure:

same file naming convention

History Monthly Latest
RR_XX_YY_CODE<_ZZZ>.nc RR_YYYYMM_XX_YY_CODE.nc RR_LATEST_XX_YY_CODE_YYYYMMDD.nc
GL_TS_MO_15001_2010daily.nc IR_200911_TS_DB_44607.nc IR_LATEST_TS_DB_44607_20091117.nc
    Common:
  • RR: region bigram
  • XX: TS (timeseries) or PR (profile)
  • YY: data type
  • CODE: platform code
    Directory-dependent:
  • Timestamp:
    YYYYMMDD (latest)& YYYYMM (monthly)
  • Fixed name: LATEST (latest)
  • _ZZZ: optional subsetting code (history)

Distribution Units provides 2 types of products :

  • Near Real Time (NRT) - for all parameters
  • Collections of data files (netCDFs) continuously updated on the Distribution Units (DUs) - Automate NRTQC

  • Reprocessed (REP) - only for some parameters
  • Delayed mode data that has been subject of a throughout Scientific assesment - visual QC.
    Build over historical data and delivered by versions, typically once a year.

NORTH WEST SHELVES
ARTIC
BALTIC
IBERIA-BISCAY-IRELAND
BLACK SEA
MED
GLOBAL
INSITU_GLO_NRT_OBSERVATIONS_013_030
INSITU_GLO_TS_REP_OBSERVATIONS_013_001_b
INSITU_GLO_UV_L2_REP_OBSERVATIONS_013_044
INSITU_GLO_TS_OA_NRT_OBSERVATIONS_013_002_a
INSITU_GLO_TS_OA_REP_OBSERVATIONS_013_002_b
GLOBAL_ANALYSIS_PHYS_001_020
GLOBAL_REP_PHY_001_021

HOW TO ACCESS
IN SITU TAC DATA PRODUCTS?

MANDATORY

1. GET GRANTED

CMEMS user & password at http://marine.copernicus.eu/services-portfolio/register-now/
http://marine.copernicus.eu/services-portfolio/register-now/

2. GET YOUR DATA

CMEMS Web Portal

(Interactive)

All CMEMS products are available via its Catalogue at: http://marine.copernicus.eu/services-portfolio/access-to-products/


In Situ TAC products are no exception to the rule.

Catalogue

INSTAC Web Portal

(Interactive)

All INSTAC products are listed at: http://www.marineinsitu.eu/products/ with shorcuts to the CMEMS Catalogue

Latest NRT data - INSTAC Dashboard at
http://www.marineinsitu.eu/dashboard/

With useful KPIs
http://www.marineinsitu.eu/monitoring/

Motu Client

(Programmatically)

CMEMS DESK SERVICE:
Pratical Session 'Python and WebServices'

  • on Tuesday AM, the 26th
  • on Wednesday PM, the 27th (14:00 to 15:30 pm in the room Lisboa)

virtual machine ( ✔ techical requirements)

Guideline:
  • Can I automate downloads of CMEMS products?
  • How to write and run the script to download CMEMS products?
  • Is a download request limited in terms of data size?

See you there!!!

Index Files
&
DUs FTP

(Programmatically)

Recap!: Do you remember In Situ DUs? In Situ DUs were the responsible of distributing In Situ data products on ftps keeping exactly the same:

  • data format (netCDF - OceanSites 1.2)
  • data flagging convention
  • ftp structure
  • file naming convention

in order to facilitate selection and processing to final users.

Recap!: Do you remember In Situ DUs FTP structure?

Examples... LATEST from MED DU

#Title : in-situ observations catalog
#Description : catalog of available in-situ observations per platform
#Project : Copernicus
#Format Version : 1.0
#Last Update : 20170925110012
#catalog_id,file_name,geospatial_lat_min,geospatial_lat_max,geospatial_lon_min,geospatial_lon_max,time_coverage_start,time_coverage_end,provider,date_update,data_mode,parameters
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_3901853_20170826.nc,34.82809,34.82809,23.86541,23.86541,2017-08-26T11:55:00Z,2017-08-26T11:55:00Z,Institute of Marine Sciences Middle East Technical University (IMS-METU),2017-08-28T18:05:57Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_3901907_20170826.nc,40.68048,40.68048,5.48433,5.48433,2017-08-26T02:48:20Z,2017-08-26T02:48:20Z,Euro-Argo,2017-08-28T18:06:02Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_3901978_20170826.nc,41.16919,41.16919,17.58031,17.58031,2017-08-26T07:58:30Z,2017-08-26T07:58:30Z,Euro-Argo,2017-08-28T19:06:48Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6900636_20170826.nc,38.37700,38.37700,5.40800,5.40800,2017-08-26T00:21:06Z,2017-08-26T00:21:06Z,ICM Institut de Ciencies del Mar - Barcelona,2017-08-27T04:07:20Z,R, PRES TEMP PRES_ADJUSTED PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901249_20170826.nc,39.72242,39.72242,1.69070,1.69070,2017-08-26T06:05:00Z,2017-08-26T06:05:00Z,SOCIB (Spain),2017-08-28T19:16:56Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901250_20170826.nc,39.20740,39.20740,4.77373,4.77373,2017-08-26T06:12:00Z,2017-08-26T06:12:00Z,SOCIB (Spain),2017-08-28T19:16:58Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901773_20170826.nc,33.11633,33.11633,31.46497,31.46497,2017-08-26T09:37:00Z,2017-08-26T09:37:00Z,,2017-08-26T13:05:36Z,R, PRES PSAL TEMP DOX2 DOWNWELLING_PAR DOWN_IRRADIANCE490 DOWN_IRRADIANCE380 DOWN_IRRADIANCE412 CDOM BBP700 CPHL CPHL_ADJUSTED NTAW
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901823_20170826.nc,38.85421,38.85421,10.69752,10.69752,2017-08-26T00:01:00Z,2017-08-26T00:01:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-27T07:07:27Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901824_20170826.nc,32.67120,32.67120,24.85794,24.85794,2017-08-26T23:58:00Z,2017-08-26T23:58:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-28T19:17:03Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901839_20170826.nc,40.11784,40.11784,3.33751,3.33751,2017-08-26T00:01:00Z,2017-08-26T00:01:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-27T07:07:31Z,R, PRES PRES_ADJUSTED TEMP TEMP_ADJUSTED PSAL PSAL_ADJUSTED
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901846_20170826.nc,34.44917,34.44917,18.52256,18.52256,2017-08-26T23:57:00Z,2017-08-26T23:57:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-28T19:17:08Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6901862_20170826.nc,39.65252,39.65252,17.89844,17.89844,2017-08-26T10:33:00Z,2017-08-26T10:33:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-26T15:13:31Z,R, PRES PSAL TEMP DOWNWELLING_PAR DOWN_IRRADIANCE490 DOWN_IRRADIANCE380 DOWN_IRRADIANCE412 CDOM BBP700 CPHL CPHL_ADJUSTED
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6902688_20170826.nc,39.73100,39.73100,7.05200,7.05200,2017-08-26T06:05:00Z,2017-08-26T06:05:00Z,,2017-08-27T19:16:37Z,R, PRES TEMP PSAL DOX2 MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6902764_20170826.nc,38.01364,38.01364,1.39542,1.39542,2017-08-26T04:58:20Z,2017-08-26T04:58:20Z,SHOM Service Hydrographique et Oceanographique de la Marine,2017-08-28T19:17:13Z,R, PRES PSAL TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6902770_20170826.nc,32.78047,32.78047,28.96149,28.96149,2017-08-26T05:02:20Z,2017-08-26T05:02:20Z,SHOM Service Hydrographique et Oceanographique de la Marine,2017-08-28T19:17:20Z,R, PRES PSAL TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6903203_20170826.nc,35.73067,35.73067,21.86430,21.86430,2017-08-26T05:59:00Z,2017-08-26T05:59:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-28T19:17:25Z,R, PRES TEMP PSAL DOX2
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6903224_20170826.nc,38.99700,38.99700,20.04400,20.04400,2017-08-26T06:58:00Z,2017-08-26T06:58:00Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-08-27T17:06:44Z,R, PRES TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_PR_PF_6903225_20170826.nc,39.65874,39.65874,15.67174,15.67174,2017-08-26T02:40:16Z,2017-08-26T02:40:16Z,OGS Istituto Nazionale di Oceanografia e di Geofisica Sperimentale,2017-09-02T05:06:27Z,R, PRES TEMP PSAL DOX2
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/latest/20170826/GL_LATEST_TS_DB_6100757_20170826.nc,34.92120,35.05840,19.53440,19.70740,2017-08-26T00:00:00Z,2017-08-26T23:00:00Z,Unknown institution,2017-08-27T04:36:06Z,R, DEPH ATMS TEMP PSAL

Examples... MONTHLY from MED DU

#Title : in-situ observations catalog
#Description : catalog of available in-situ observations per platform
#Project : Copernicus
#Format Version : 1.0
#Last Update : 20170925110014
#catalog_id,file_name,geospatial_lat_min,geospatial_lat_max,geospatial_lon_min,geospatial_lon_max,time_coverage_start,time_coverage_end,provider,date_update,data_mode,parameters

MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/GL_201201_TS_DB_61787.nc,37.9933,38.2,6.6971,13.4556,2012-01-01T01:00:00Z,2012-01-15T10:00:00Z,IFREMER,2013-04-06T05:06:17Z,R, TEMP EWCT NSCT DEPH
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/GL_201201_TS_DB_61788.nc,38.55600,39.87450,3.27580,4.76350,2012-01-01T00:00:19Z,2012-01-31T23:30:17Z,,2017-07-29T15:17:47Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/GL_201201_TS_DB_61792.nc,39.1625,39.173,-0.24845,0.1677,2012-01-01T00:02:11Z,2012-01-31T23:30:00Z,IFREMER,2013-04-06T05:49:04Z,R, TEMP EWCT NSCT DEPH
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/GL_201201_TS_DB_61793.nc,36.06870,37.32010,-1.41450,0.18100,2012-01-01T00:01:41Z,2012-01-31T23:30:22Z,,2017-07-29T15:17:49Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/GL_201201_TS_DB_61850C.nc,36.882,36.883,10.767,10.767,2012-01-01T17:30:00Z,2012-01-02T19:30:00Z,IFREMER,2013-04-06T04:50:56Z,R, TEMP DEPH
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/MO_201201_TS_DC_108743.nc,36.83715,37.62300,19.65071,21.08360,2012-01-01T00:00:00Z,2012-01-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/MO_201201_TS_DC_300234010168380.nc,40.96201,42.08307,9.75803,11.64286,2012-01-01T00:00:00Z,2012-01-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/MO_201201_TS_DC_300234010652970.nc,39.11646,39.46365,8.05844,8.35325,2012-01-01T00:00:00Z,2012-01-06T03:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201201/MO_201201_TS_DC_3700572.nc,37.72119,38.24943,2.22189,4.17362,2012-01-01T00:00:00Z,2012-01-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201202/GL_201202_TS_DB_61788.nc,37.89860,38.58310,1.16660,3.30370,2012-02-01T00:00:16Z,2012-02-29T23:30:39Z,,2017-07-29T15:17:47Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201202/GL_201202_TS_DB_61793.nc,36.21110,37.13710,-0.68700,7.18730,2012-02-01T00:00:47Z,2012-02-28T10:00:00Z,,2017-07-29T15:17:50Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201202/MO_201202_TS_DC_108743.nc,36.93371,38.78457,18.91089,20.67823,2012-02-01T00:00:00Z,2012-02-29T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201202/MO_201202_TS_DC_300234010168380.nc,39.79373,41.81726,9.98525,11.24973,2012-02-01T00:00:00Z,2012-02-29T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201202/MO_201202_TS_DC_3700572.nc,36.84541,37.79088,2.30323,6.96606,2012-02-01T00:00:00Z,2012-02-29T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201202/MO_201202_TS_DC_dep0002-scb-idrifter003.nc,39.89922,40.06210,2.61684,2.99964,2012-02-28T12:00:00Z,2012-02-29T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:04:59Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61788.nc,37.62600,38.42090,-0.10480,1.17130,2012-03-01T00:00:15Z,2012-03-31T23:30:16Z,,2017-07-29T15:17:47Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61791.nc,41.10780,41.10800,1.23870,1.23900,2012-03-01T00:00:14Z,2012-03-31T23:30:16Z,,2017-07-29T15:17:48Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61792.nc,39.16711,39.16712,-0.24312,-0.24311,2012-03-01T00:30:00Z,2012-03-31T21:32:10Z,,2017-07-29T15:17:48Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61850.nc,42.73670,44.11690,8.94430,9.85620,2012-03-19T13:30:00Z,2012-03-31T23:30:35Z,,2017-07-29T15:17:50Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61851.nc,42.50940,43.34910,6.13350,7.48510,2012-03-21T05:30:00Z,2012-03-31T23:30:31Z,,2017-07-29T15:21:02Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61852.nc,42.73700,43.90940,9.52010,9.86880,2012-03-19T13:30:00Z,2012-03-31T23:30:18Z,,2017-07-29T15:21:03Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61853.nc,42.70380,44.17870,9.26340,9.76460,2012-03-05T14:34:14Z,2012-03-31T23:30:12Z,,2017-07-29T15:21:04Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61854.nc,42.66900,43.89540,9.31450,10.25670,2012-03-05T13:30:00Z,2012-03-31T23:30:14Z,,2017-07-29T15:21:05Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61855.nc,42.69090,43.67260,9.48490,9.89610,2012-03-05T13:30:00Z,2012-03-31T23:30:18Z,,2017-07-29T15:21:05Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61856.nc,42.86750,43.61500,5.11750,7.62400,2012-03-21T11:00:00Z,2012-03-31T23:30:10Z,,2017-07-29T15:21:06Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61857.nc,42.85830,43.60160,6.41980,7.74130,2012-03-21T12:04:14Z,2012-03-31T23:30:29Z,,2017-07-29T15:21:07Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61858.nc,43.59090,43.59090,7.23210,7.23210,2012-03-28T16:04:14Z,2012-03-31T08:34:14Z,,2017-07-29T15:21:07Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61859.nc,43.20030,43.55190,6.72160,7.27450,2012-03-21T05:30:00Z,2012-03-31T23:31:00Z,,2017-07-29T15:21:06Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61860.nc,42.53490,43.48000,4.32500,7.33710,2012-03-21T05:30:00Z,2012-03-31T23:30:17Z,,2017-07-29T15:24:15Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61861.nc,42.89770,43.68760,5.87010,7.57790,2012-03-21T11:00:00Z,2012-03-31T23:30:17Z,,2017-07-29T15:24:16Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61935.nc,42.88260,43.58270,5.00580,7.50520,2012-03-07T10:00:00Z,2012-03-31T23:30:07Z,,2017-07-29T15:24:16Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61937.nc,43.60120,43.68850,7.15770,7.38170,2012-03-21T02:00:22Z,2012-03-31T23:30:19Z,,2017-07-29T15:24:16Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61938.nc,43.11600,43.60270,6.37220,7.54890,2012-03-07T10:00:00Z,2012-03-31T23:30:05Z,,2017-07-29T15:24:19Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61939.nc,42.98460,43.41540,6.95900,7.42460,2012-03-21T05:30:00Z,2012-03-31T23:31:07Z,,2017-07-29T15:24:19Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61940.nc,43.39040,43.42460,7.68700,7.78410,2012-03-20T13:30:00Z,2012-03-28T17:30:04Z,,2017-07-29T15:24:18Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61941.nc,42.95220,43.56450,5.70760,7.42280,2012-03-07T10:00:00Z,2012-03-31T23:30:26Z,,2017-07-29T15:24:18Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/GL_201203_TS_DB_61942.nc,42.75940,43.63150,4.11760,7.58620,2012-03-07T10:00:00Z,2012-03-31T23:30:25Z,,2017-07-29T15:27:31Z,R, DEPH TEMP NSCT EWCT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/MO_201203_TS_DC_108743.nc,37.73371,37.77347,19.08843,19.12160,2012-03-01T00:00:00Z,2012-03-01T06:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:05:00Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/MO_201203_TS_DC_300234010168380.nc,40.58330,41.95284,9.57632,11.48234,2012-03-01T00:00:00Z,2012-03-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:05:00Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/MO_201203_TS_DC_3700572.nc,37.33028,38.66776,6.28530,8.97993,2012-03-01T00:00:00Z,2012-03-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:05:00Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/MO_201203_TS_DC_6100850.nc,42.75228,44.10556,8.96500,9.85536,2012-03-21T03:00:00Z,2012-03-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:05:00Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/MO_201203_TS_DC_6100851.nc,42.52947,43.33353,6.13354,7.48251,2012-03-21T15:00:00Z,2012-03-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:05:00Z,D, EWCT NSCT DEPH DROGUE_PRESENCE
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/monthly/drifter/201203/MO_201203_TS_DC_6100852.nc,42.75580,43.90901,9.53587,9.86809,2012-03-21T03:00:00Z,2012-03-31T21:00:00Z,OGS (Istituto Nazionale di Oceanografia e di Geofisica Sperimentale),2017-08-30T15:05:00Z,D, EWCT NSCT DEPH DROGUE_PRESENCE

Examples... HISTORY from MED DU

#Title : in-situ observations catalog
#Description : catalog of available in-situ observations per platform
#Project : Copernicus
#Format Version : 1.0
#Last Update : 20170925110015
#catalog_id,file_name,geospatial_lat_min,geospatial_lat_max,geospatial_lon_min,geospatial_lon_max,time_coverage_start,time_coverage_end,provider,date_update,data_mode,parameters

MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_PR_TE_61501.nc,42.12,42.82,5.6,7.4,1999-09-10T08:58:00Z,1999-11-15T06:48:00Z,MEDS,2017-02-02T04:50:47Z,R, PRES TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_PR_TE_61502.nc,41.76,42.01,3.67,3.79,1999-09-10T07:42:00Z,1999-10-04T10:36:00Z,MEDS,2017-02-02T04:50:47Z,R, PRES TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_PR_TE_61504.nc,42.18,43.45,6.15,7.62,1999-09-10T07:42:00Z,1999-11-11T14:37:00Z,MEDS,2017-02-02T04:50:48Z,R, PRES TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_PR_TE_61537.nc,41.15,41.34,3.77,4.13,1999-12-13T16:56:00Z,1999-12-15T05:30:00Z,US NAVAL OCEANOGRAPHIC OFFICE(BAY ST LOUIS MISSISSIPI),2017-02-02T04:50:48Z,R, PRES TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100536.nc,40.67270,42.19970,7.23520,7.67410,2015-07-01T00:00:00Z,2015-07-28T20:00:00Z,Unknown institution,2017-07-13T14:52:30Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100537.nc,38.91750,39.05520,1.11350,1.22300,2015-07-01T00:00:00Z,2015-07-19T06:00:00Z,Unknown institution,2017-07-13T14:52:30Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100538.nc,37.36790,39.92880,5.12680,7.26450,2015-07-01T00:00:00Z,2015-10-13T06:00:00Z,Unknown institution,2017-07-13T14:52:31Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100539.nc,37.03650,37.71640,-1.08950,0.92860,2015-07-01T00:00:00Z,2015-08-18T05:00:00Z,Unknown institution,2017-07-13T14:52:31Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100863.nc,39.97820,42.57650,0.62040,3.96730,2015-08-19T21:00:00Z,2015-10-13T06:00:00Z,Unknown institution,2017-07-13T14:52:35Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100864.nc,36.52040,40.09410,7.36450,11.22990,2015-08-19T21:00:00Z,2015-09-23T09:00:00Z,Unknown institution,2017-07-13T14:52:35Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100865.nc,40.17630,41.11510,2.94810,4.08950,2015-08-20T00:00:00Z,2015-10-13T06:00:00Z,Unknown institution,2017-07-13T14:52:36Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100866.nc,39.96680,40.63280,5.04820,7.15950,2015-08-20T01:00:00Z,2015-10-13T05:00:00Z,Unknown institution,2017-07-13T14:52:41Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_100867.nc,42.41490,43.72860,3.60250,8.72870,2015-08-19T23:00:00Z,2015-10-13T06:00:00Z,Unknown institution,2017-07-13T14:52:41Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_1200518.nc,31.49640,34.60820,27.04060,32.00920,2017-05-24T12:53:00Z,2017-08-22T18:00:00Z,Unknown institution,2017-09-01T15:18:07Z,R, DEPH ATMS TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_12506.nc,25.94000,28.06200,34.79800,35.84900,2004-12-08T19:00:00Z,2005-03-07T19:00:00Z,Naval Oceanographic Office (NAVO),2017-07-13T14:57:24Z,R, DEPH TEMP ATMP ATPT ATMS
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_1300574.nc,36.79732,37.13324,1.24871,3.70004,2016-03-09T14:45:00Z,2016-03-22T09:59:00Z,Unknown institution,2017-07-13T15:07:10Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_1300905.nc,32.27688,39.61784,-2.65949,16.06133,2015-10-13T09:00:00Z,2016-08-16T17:00:00Z,Atlantic Oceanographic and Meteorological Laboratory (AOML) National Oceanic and Atmospheric Administration (NOAA),2017-07-13T15:07:14Z,R, DEPH TEMP
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13527.nc,1.26800,37.78300,-61.64000,8.88800,2004-10-23T00:30:00Z,2015-10-12T23:57:00Z,Unknown institution,2017-07-13T15:11:42Z,R, DEPH TEMP ATMP ATMS ATPT
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13566.nc,8.75600,40.05600,-47.07300,2.99600,2009-08-19T20:01:00Z,2016-01-12T10:44:00Z,Unknown institution,2017-07-13T15:11:46Z,R, DEPH TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13574.nc,10.67900,38.74600,-88.81300,1.78600,2009-11-25T12:59:00Z,2016-03-08T22:51:00Z,Unknown institution,2017-07-13T15:29:44Z,R, DEPH TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13586.nc,11.32800,38.11300,-68.44500,5.99900,2005-06-05T08:54:00Z,2016-01-26T14:27:00Z,Meteo-France / Centre de Meteorologie Marine (CMM),2017-07-13T15:29:55Z,R, DEPH TEMP PSAL
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13640.nc,11.84100,36.83500,-57.42200,-5.39900,2006-07-13T15:59:00Z,2015-10-12T23:57:00Z,Unknown institution,2017-07-13T15:42:59Z,R, DEPH TEMP ATMP ATPT ATMS
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13904.nc,-0.54400,38.10600,-74.16700,3.01300,2006-06-22T14:40:00Z,2014-10-23T20:00:00Z,Atlantic Oceanographic and Meteorological Laboratory (AOML)
National Oceanic and Atmospheric Administration (NOAA),2017-07-13T15:43:02Z,R, DEPH TEMP ATMP ATPT ATMS
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13905.nc,13.76800,44.31600,-76.04300,9.63600,2006-06-28T12:23:00Z,2015-10-12T23:00:00Z,Unknown institution,2017-07-13T15:43:02Z,R, DEPH TEMP ATMP ATPT ATMS
MYO-MOON-01,ftp://medinsitu.hcmr.gr/Core/INSITU_MED_NRT_OBSERVATIONS_013_035/history/drifter/GL_TS_DB_13906.nc,11.80100,41.91700,-74.50200,4.63900,2006-06-28T05:25:00Z,2014-11-10T18:00:00Z,Scripps Institution of Oceanography La Jolla (SIO),2017-07-13T15:43:02Z,R, DEPH TEMP ATMP ATPT ATMS

1. Let's connect to a DU ftp and download the index file!

2. Let's go through the index file and download netCDFs!

3. Let's schedule it

Donwload Anaconda

https://www.anaconda.com/download/

On the right corner select: New Pyhton 3

Import the libraries we are going to use

import ftplib
import numpy as np

Download the index file of a given DU

user = '' #type CMEMS user name
password = '' #type CMEMS password
product_name = '' #type aimed CMEMS in situ product
distribution_unit = '' #type aimed hosting institution
index_file = '' #type aimed index file

ftp=ftplib.FTP(distribution_unit,user,password)
ftp.cwd("Core")
ftp.cwd(product_name)
aimedFileName = index_file
local_filename = aimedFileName
local_file = open(local_filename, 'wb')
ftp.retrbinary('RETR ' + aimedFileName, local_file.write)
local_file.close()
ftp.quit()
#ready when 221 Goodbye.!

Go through the index file and make your selection!

index = np.genfromtxt(index_file, skip_header=6,
unpack=False, delimiter=',', dtype=None,
names=['catalog_id', 'file_name',
'geospatial_lat_min', 'geospatial_lat_max', 'geospatial_lon_min', 'geospatial_lon_max','
time_coverage_start', 'time_coverage_end', 'provider', 'date_update', 'data_mode', 'parameters'
])

Example: selection by parameter - metadata

#read file lines (iterate over them)
for entry in index:
file_name = entry['file_name'].decode('utf-8')
parameters = entry['parameters'].decode('utf-8')
param_list = parameters.split(' ');
#set up a selection criteria: i.e specific parameter
if 'PSAL' in param_list:
last_idx_slash = file_name.rfind('/')
ncdf_file_name = file_name[last_idx_slash+1:]
folders = file_name.split('/')[3:len(file_name.split('/'))-1]
host = file_name.split('/')[2] #distribution unit
ftp=ftplib.FTP(host,user,password)
for folder in folders:
ftp.cwd(folder)
local_file = open(ncdf_file_name, 'wb')
ftp.retrbinary('RETR '+ncdf_file_name, local_file.write)
local_file.close()
ftp.quit()

Example: selection by data_types - name_convention

#read file lines (iterate over them)
for entry in index:
file_name = entry['file_name'].decode('utf-8')
last_idx_slash = file_name.rfind('/')
ncdf_file_name = file_name[last_idx_slash+1:]
#history: position 2
#monthly and latest: position 3
data_type = ncdf_file_name.split('_')[position]
aimed_data_type = ''#choose a data_type
#set up a selection criteria: i.e specific datatype
if data_type == aimed_data_type :
folders = file_name.split('/')[3:len(file_name.split('/'))-1]
host = file_name.split('/')[2] #distribution unit
ftp=ftplib.FTP(host,user,password)
for folder in folders:
ftp.cwd(folder)
local_file = open(ncdf_file_name, 'wb')
ftp.retrbinary('RETR '+ncdf_file_name, local_file.write)
local_file.close()
ftp.quit()

Schedule the task!

    Linux o Mac?

    Rely on crontab!

    Windows?

    Rely on the equivalent task escheduler??

    microsoft

More at:
https://github.com/CopernicusMarineInsitu/INSTACTraining/

http://www.marineinsitu.eu/material/

Doubts?