System download


Installation package

The installation package of the Sen2-Agri system has been split into 3 parts to ease its download:

  • A zip archive containing all the necessary binaries and setup scripts  (not considering MACCS, see below) [~700MB]:
    • install_script – contains the installation scripts that are used to create the distribution and to install the system and the tool needed for the integration of the Sen2-Agri processors in SNAP;
    • gipp – the gipp files used by the L2A processor (MACCS) during the processing of the L1C products;
    • rpm_binaries – the RPM files for all other system components (SLURM, orchestrator, downloader, processors);
  • A zip archive containing the SRTM dataset [~16 GB]: files needed by MACCS
  • A zip archive containing the SWBD dataset [~900 MB]: files needed by MACCS

IMPORTANT: MACCS – that is used by the Sen2-Agri system for the atmospheric corrections and non-valid pixels masking – is not included in the installation package of the Sen2-Agri system and have to be downloaded separately from the CNES site. Nevertheless, MACCS must be installed before running the Sen2-Agri installer, as described in the Software User Manual or in the FAQ section;  . The Sen2-Agri installation script will look for the executable and prompt for its installed path location.

You can download the latest Software User Manual, as well synthesis techincal documents in the Technical Documents page.

Sen2-Agri system download Version 1.8.2 [Release 16-07-2018]

System requirement

If you intend to use the Sen2-Agri system on a local site (~90 000km²), the minimum system requirements to install and run this system are as follows:

  • CPU: 8 Cores
  • RAM: 32GB
  • HDD Storage: 4 TB
  • SSD Storage: 250 GB (optionnal – for temporary files)

 

If you intend to use the Sen2-Agri system on a national site (~500 000km²), the minimum system requirements to install and run this system are as follows:

  • CPU: 16 to 32 Cores
  • RAM: 64GB to 128 GB
  • HDD Storage: 10 TB is enough even if 15TB is recommended
  • SSD Storage: 1 TB (optional – for temporary files)

 

Installation/Update

How do I install the Sen2-Agri system?

If you want to configure a new server with the v1.8.2, the installation procedure is detailed on the section 3 of the Software User Manual [v2.7] .

How do I update my Sen2-Agri system?

If you already have a version of the Sen2-Agri system installed, you can easily upgrade it to a newer version by downloading the new installation package (no need to download again the srtm-swbd-and maccs) and running the update script. To do so, you should navigate to the installation package directory and simply run the update command. All version updates can be performed using the same script included in the installation package:.

cd ./Sen2AgriDistribution/install_script/
sudo ./update.sh

! Adjusted parameters for the atmospheric correction (GIPP) are available in the v1.8.2 installation package. A dedicated update script allows you to replace the old one by the new one but the procedure may vary depending on how you are currently using the Sen2-Agri system. Please look carefully at this v1.8.2. release post to identifiy you update scenario.

Changelog

System Version 1.8.2 [13/07/2018]: Public release of the version 1.8.2 of the system

Fixed
  • Updated GIPP
  • Handling of multi-polygons site extent geometry
  • Decimal precision in geometry
  • Handling of the time lag between the Sci-Hub and AWS acquisition availability (normally, this can be tuned by changing values in datasource table for max_retries and retry_interval).
Added
  •  A new configuration key was added in order to force queries from the start of the season instead of the last successfuly download date. The config key (that can be set only for passing again all seasons of the site and not for a nominal functionning) can be set to “config” table as “scheduled.lookup.all_products.enabled = true”. After a complete pass of the season is made, this key should be removed or set to false (and sen2agri-services restared) as it does not functions correctly with the near-realtime downloading mode.
  • Support for local repositories that have paths in the format 2018/1/2 (and not 2018/01/02) by setting in the sen2agri-services.properties file the following keys:
     AWSDataSource.Sentinel2.usePadding=false
     AWSDataSource.Landsat8.usePadding=false
Known issues
  • The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
  • The SAFE formatting application sometimes outputs unusable (e.g. black) previews
  • The SAFE and L2A product previews are not gamma-corrected and can be too dark
  • The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
  • The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
  • The crop type and mask training step sometimes crashes while loading the OpenCV models
  • MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
  • The MACCS launcher script does not process different tiles of the same site in parallel
  • The product formatting and tile aggregation steps are inefficient
  • Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
  • The trimming step of the Crop Mask processor still uses a large amount of memory
  • The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
  • The dashboard previews don’t match their bounds rectangle because of projection mismatch
  • The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
  • The website has display issues on Safari
  • When deleting a site, the folders for the L2A products that were not processed by MACCS are not deleted and should be deleted manually. Normally these folders contain no valid product and contain only log and EEF files.

System Version 1.8.1 [07/05/2018]: Public release of the version 1.8.1 of the system

Fixed
  • LAI Multi-Date processors are now functionning also for products obtained using the new L3B monodate algorithm.
Known issues

Please refer to the known issues of the v1.8.

System Version 1.8 [06-04-2018]: Public pre-release of the version 1.8 of the system

Added
  • The L4A/L4B processors use a platform-independent algorithm for the training/validation split. This is a step towards allowing the processor to work on other platforms and give the same results.
  • Added the sen2agri-services application that is performing the following:
    • Replaces the old downloaders. Now there are defined datasources that allow customization of sources for query and for download. For example, for S2, the query can be performed from SciHub but the download can be performed from AWS or from a local archive repository (if products are already downloaded locally)
    • Offers RESTful services for interrogating system for :
      * sites configurations, enable/disable/delete sites, retrieve the site seasons
      * enable/disable downloading for a site, for a certain satellite
      * start or stop the downloads for a site/satellite
      * retrieve or update the configuration of a datasource
      * receive notifications from the system
  • Added the possibility to delete a site from the web interface
  • Added the possibility to upload insitu and strata data from the web interface
  • Added INRA implementation for the LAI. In this moment, it works only for the Sentinel2 products and is disabled by default.
  • GUI Products tab has now the possibility of filtering products by site, season, tiles, product type or by specifying an interval
  • Option to disable L8 (downloading and processing) for a site from the IHM
  • Added the possibility to delete a scheduled job from the Dashboard in IHM
  • Added buttons for Pause/Resume/Cancel a job in the System Monitoring page of the IHM
  • When the final product is creating, a lock file is marking that the product is not complete yet.
  • Added options in the configuration to delete SRE or FRE file or to compress the outputs when MACCS creates the final product.
  • Added a new state for the downloader_history in order to know when a product is currently processing by MACCS.
  • Display version of the system in IHM website footer
Changed
  • Breaking change: the L4A and L4B processors now use a different algorithm for the trainig/validation split. Output files will be different from the ones in the previous versions. The accuracy scores will often be lower because poorly-represented classes will now be present in the validation set. They weren’t taken into account previously.
  • The SampleSelection application places all the features in both the training and the validation sets if there are too few of them and one of them would end up empty according to the configured split (e.g. 75% training with only 2 features). This allows taking them into account for validation, although with lower accuracy.
  • Removed L2A processor from the GUI Dashboard tab since it can not be launch from here
  • Changed processor names in the GUI Dashboard.
  • The downloads stops now when a site is disabled.
  • Corrected the LAI to avoid saturation of values.
  • Usage of CCI 2015 instead of CCI 2010.
Fixed
  • Fixed crash in the SampleSelection application used by the L4 processors when a feature with no geometry is present
  • The SampleSelection application always puts features in one of the training or validation set. Previously, they were sometimes lost.
  • Composite Python Processor was updated to generate both 10 and 20 m resolutions.
  • Corrections in LAI and Pheno NDVI processors Python scripts.
Known issues
  • The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
  • The SAFE formatting application sometimes outputs unusable (e.g. black) previews
  • The SAFE and L2A product previews are not gamma-corrected and can be too dark
  • The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
  • The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
  • The crop type and mask training step sometimes crashes while loading the OpenCV models
  • MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
  • The MACCS launcher script does not process different tiles of the same site in parallel
  • The product formatting and tile aggregation steps are inefficient
  • Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
  • The trimming step of the Crop Mask processor still uses a large amount of memory
  • The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
  • The dashboard previews don’t match their bounds rectangle because of projection mismatch
  • The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
  • The website has display issues on Safari

System Version 1.7 [30-11-2017]: Second public version of the system

 

### Added
– The system is now able to download and process Sentinel-2B images.
– The unsupervised crop mask processor can now use (optionally, on by default) the red edge bands
– Added the local mode in the Sentinel-2 downloader in order to use a local L1C repositories as products store
– Added a new flag for L1C products processing in order to determine if the product was handled OK or not by MACCS.

### Changed
– Products in the `RT` and `T2` Landsat 8 collections are no longer downloaded
– Changed the crop mask post-filtering to ignore negative reflectance values
– Reduced the CPU usage of the crop mask post-filtering NDVI extraction step by 20%, and the wall clock time by 15% when running with 4 threads
– Reduced the CPU and disk usage by merging the crop mask post-filtering no data filling and PCA steps; CPU usage went down by 27% and disk usage by about 20 GB for a single tile
– The Landsat 8 downloader no longer keep a download status line if it’s not writing to a TTY

### Fixed
– Fixed an issue that made the crop mask post-filtering output invalid rasters when both Sentinel-2 and Landsat 8 products were used for post-filtering; this was disabled by default
– Changed the post-filtered crop mask data type from `Float32` to `Int16`
– Fixed an issue that caused the supervised crop mask processor to crash on tiles spanned by multiple strata
– Fixed an issue that sometimes caused MACCS to be passed an L3C product instead of the previous L2A one
– Fixed a potential division by zero in the Landsat 8 downloader

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The MACCS launcher script does not process different tiles of the same site in parallel
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

System version 1.6.1 [05-07-2017]: First public release of the system

### Added
– Landsat 8 collections support for the downloader

### Fixed
– The Landsat 8 downloader now properly reports USGS authentication errors when using a proxy
– Fixed an issue in the retrieval of season dates for sites with more than one season

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step eagerly flags products with a tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

System version 1.6 [08-06-2017]: Pre-release of the system to Champion-users

### Added
– The possibility to select the input bands when creating the LAI monodate products, including 20M bands and possibility to use or not NDVI and RVI.
– Scripts for deleting a site and for filtering a site tiles
– Multiple seasons in the UI
– The website monitoring page can now display the command line of a job and its output
– The supervised crop mask processor can now use (optionally, on by default) features derived from the red edge bands

### Changed
– In the LAI scheduled jobs now are taken into account the insertion dates instead of product creation date.
– Limited the crop type processor training to `10 000` samples / tile
– The crop type and supervised crop mask processors now include the in-situ data in the output product
– The short name and extent of sites can no longer be updated from the website
– The website monitoring page now sorts jobs by submission time instead of end time
– The installer now sets `executor.listen-ip` to `127.0.0.1` and the SLURM node name to `localhost`
– The binaries were recompiled for `GEOS 1.5`
– Increased crop mask and crop type tile thread hint from `2` to `5` to reduce the risk of out-of-memory conditions

### Fixed
– Changed the advanced mode for processors parameters in the sen2agri-config configuration.
– In IPP file name for L3A is not inserted the used bands_mapping and not the one from the DB.
– The missing L8 HDR from the L3C/L3D IPP file
– Corrected the date order in the creation of L3C/L3D products.
– Fixed a crash in the crop type processor when one of the tiles was missing Landsat 8 data
– Website custom L4A jobs use a value of `40 000` training samples / tile instead of `4 000` to match the processor default

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step eagerly flags products with a tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

License

This program (Sentinel-2 for Agriculture system) is a free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3 of the License.