System download


Installation package


The installation package of the Sen2-Agri system version 1.7 [release on 30/11/2017] the installation package has been split into 3 parts to ease its download.

  • A zip archive containing all the necessary binaries and setup scripts  (not considering MACCS [1]):
    • install_script – contains the installation scripts that are used to create the distribution and to install the system and the tool needed for the integration of the Sen2-Agri processors in SNAP;
    • gipp – the gipp files used by the L2A processor (MACCS) during the processing of the L1C products;
    • rpm_binaries – the RPM files for all other system components (SLURM, orchestrator, downloader, processors);
  • A zip archive containing the SRTM dataset (~16 GB): files needed by MACCS
  • A zip archive containing the SWBD dataset (~900 MB): files needed by MACCS

IMPORTANT: MACCS – that is used by the Sen2-Agri system for the atmospheric corrections and non-valid pixels masking – is not included in the installation package of the Sen2-Agri system and have to be downloaded separately from the CNES site. Nevertheless, MACCS must be installed before running the Sen2-Agri installer, as described in the Software User Manual or in the FAQ section;  . The Sen2-Agri installation script will look for the executable and prompt for its installed path location.

You can download the latest Software User Manual, as well synthesis documents in the Technical Documents page.

Sen2-Agri system download Version 1.7 [Release 30/11/2017]

System requirement

If you intend to use the Sen2-Agri system on a local site (~90 000km²), the minimum system requirements to install and run this system are as follows:

  • CPU: 8 Cores
  • RAM: 32GB
  • HDD Storage: 4 TB
  • SSD Storage: 250 GB (optionnal – for temporary files)

 

If you intend to use the Sen2-Agri system on a national site (~500 000km²), the minimum system requirements to install and run this system are as follows:

  • CPU: 16 to 32 Cores
  • RAM: 64GB
  • HDD Storage: 10 TB is enough even if 15TB is recommended
  • SSD Storage: 1 TB (optionnal – for temporary files)

 

Installation/Update

How do I install the Sen2-Agri system?

The installation procedure is detailed on the section 3 of the Software User Manual [v2.5] .

How do I update the Sen2-Agri system?

In case you have already a version of the Sen2-Agri system installed, you can easily upgrade it to a newer version by downloading the new installation package and running the update script. To do so, you should navigate to the installation package directory and run the update command.

cd ./Sen2AgriDistribution/install_script/

 

sudo ./update.sh

Changelog

System Version 1.7 [30-11-2017]: Second public version of the system

### Added
– The system is now able to download and process Sentinel-2B images.
– The unsupervised crop mask processor can now use (optionally, on by default) the red edge bands
– Added the local mode in the Sentinel-2 downloader in order to use a local L1C repositories as products store
– Added a new flag for L1C products processing in order to determine if the product was handled OK or not by MACCS.

### Changed
– Products in the `RT` and `T2` Landsat 8 collections are no longer downloaded
– Changed the crop mask post-filtering to ignore negative reflectance values
– Reduced the CPU usage of the crop mask post-filtering NDVI extraction step by 20%, and the wall clock time by 15% when running with 4 threads
– Reduced the CPU and disk usage by merging the crop mask post-filtering no data filling and PCA steps; CPU usage went down by 27% and disk usage by about 20 GB for a single tile
– The Landsat 8 downloader no longer keep a download status line if it’s not writing to a TTY

### Fixed
– Fixed an issue that made the crop mask post-filtering output invalid rasters when both Sentinel-2 and Landsat 8 products were used for post-filtering; this was disabled by default
– Changed the post-filtered crop mask data type from `Float32` to `Int16`
– Fixed an issue that caused the supervised crop mask processor to crash on tiles spanned by multiple strata
– Fixed an issue that sometimes caused MACCS to be passed an L3C product instead of the previous L2A one
– Fixed a potential division by zero in the Landsat 8 downloader

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The MACCS launcher script does not process different tiles of the same site in parallel
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

System version 1.6.1 [05-07-2017]: First public release of the system

### Added
– Landsat 8 collections support for the downloader

### Fixed
– The Landsat 8 downloader now properly reports USGS authentication errors when using a proxy
– Fixed an issue in the retrieval of season dates for sites with more than one season

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step eagerly flags products with a tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

System version 1.6 [08-06-2017]: Pre-release of the system to Champion-users

### Added
– The possibility to select the input bands when creating the LAI monodate products, including 20M bands and possibility to use or not NDVI and RVI.
– Scripts for deleting a site and for filtering a site tiles
– Multiple seasons in the UI
– The website monitoring page can now display the command line of a job and its output
– The supervised crop mask processor can now use (optionally, on by default) features derived from the red edge bands

### Changed
– In the LAI scheduled jobs now are taken into account the insertion dates instead of product creation date.
– Limited the crop type processor training to `10 000` samples / tile
– The crop type and supervised crop mask processors now include the in-situ data in the output product
– The short name and extent of sites can no longer be updated from the website
– The website monitoring page now sorts jobs by submission time instead of end time
– The installer now sets `executor.listen-ip` to `127.0.0.1` and the SLURM node name to `localhost`
– The binaries were recompiled for `GEOS 1.5`
– Increased crop mask and crop type tile thread hint from `2` to `5` to reduce the risk of out-of-memory conditions

### Fixed
– Changed the advanced mode for processors parameters in the sen2agri-config configuration.
– In IPP file name for L3A is not inserted the used bands_mapping and not the one from the DB.
– The missing L8 HDR from the L3C/L3D IPP file
– Corrected the date order in the creation of L3C/L3D products.
– Fixed a crash in the crop type processor when one of the tiles was missing Landsat 8 data
– Website custom L4A jobs use a value of `40 000` training samples / tile instead of `4 000` to match the processor default

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step eagerly flags products with a tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari