System download

If you intend to use the Sen2-Agri system on a local site (~90 000km²), the minimum system requirements to install and run this system are as follows:

• CPU: 8 Cores

• RAM: 32GB

• HDD Storage: 4 TB

• SSD Storage: 250 GB (optionnal – for temporary files)

If you intend to use the Sen2-Agri system on a national site (~500 000km²), the minimum system requirements to install and run this system are as follows:

• CPU: 16 to 32 Cores

• RAM: 64GB to 128 GB

• HDD Storage: 10 TB is enough even if 15TB is recommended

• SSD Storage: 1 TB (optional – for temporary files)

How do I install the Sen2-Agri system?

If you want to configure a new server with the v2.0.3, the installation procedure is detailed on the section 3 of the Software User Manual.

How do I update my Sen2-Agri system?

The procedure to update Sen2-Agri system in v2.0.3 depends on the version of the system that you are currently running. Instructions are given in the v2.0.3 release post. We strongly recommend you reading carefully this post and following the instructions.

System Version 2.0.3 [24/11/2020]: Public release of the version 2.0.3 of the system

Added

• Statistics tab to get info about acquisition and preprocessing status
• Retriable queries (json persistence)
• Optimisation of thread usage when fetch_mode is SymLink or Check

Changed

• Move of the system database (PostgreSQL server) in a docker container
• Set the SciHub datasource auto.uncompress flag to true in order to avoid issues when Sen2Cor L2A products are downloaded from SciHub
• Extracted the L3C and L3D processors from the L3B processor (they are distinct processors now)
• Renamed in the database the processor l3b instead of l3b_lai and l3e instead of l3e_pheno (this might result in additional output directories for these products for each site).

Fixed

• Updates for the new changes in USGS API (will change again in Dec/Jan)

Known issues

• The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
• The SAFE formatting application sometimes outputs unusable (e.g. black) previews
• The SAFE and L2A product previews are not gamma-corrected and can be too dark
• The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
• The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
• The crop type and mask training step sometimes crashes while loading the OpenCV models
• MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
• The product formatting and tile aggregation steps are inefficient
• Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
• The trimming step of the Crop Mask processor still uses a large amount of memory
• The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
• The dashboard previews don’t match their bounds rectangle because of projection mismatch
• The website has display issues on Safari
• When deleting a site, the folders for the L2A products that were not processed by MACCS are not deleted and should be deleted manually. Normally these folders contain no valid product and contain only log and EEF files.

System Version 2.0.2 [05/03/2020]: Public release of the version 2.0.2 of the system

Added

• Script for importing THEIA products
• Support for using download and usage of Sen2Cor L2A products (configurable via database)
• Support for FAPAR and FCOVER production after switching to INRA algorithm in L3B processor

Changed

• The MAJA training interval was changed to 2 months instead of 3 months
• Changed to INRA version algorithm as the default in LAI production
• The option to install MACCS was removed and MAJA is considered by default
• The dialog for uploading insitu files
• The icons in the output of the jobs from the “monitoring” tab were updated to reflect more step states
• L3B automatic scheduling was changed from 10 days to 1 day, ensuring each time that all the previous L2A products before the current scheduled date were processed into L3B products

Fixed

• Updates for the new changes in USGS API
• Corrections for avoiding skipping one product per page in SciHub

Known issues

• The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
• The SAFE formatting application sometimes outputs unusable (e.g. black) previews
• The SAFE and L2A product previews are not gamma-corrected and can be too dark
• The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
• The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
• The crop type and mask training step sometimes crashes while loading the OpenCV models
• MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
• The product formatting and tile aggregation steps are inefficient
• Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
• The trimming step of the Crop Mask processor still uses a large amount of memory
• The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
• The dashboard previews don’t match their bounds rectangle because of projection mismatch
• The website has display issues on Safari
• When deleting a site, the folders for the L2A products that were not processed by MACCS are not deleted and should be deleted manually. Normally these folders contain no valid product and contain only log and EEF files

System Version 2.0.1 [10/07/2019]: Public release of the version 2.0.1 of the system

Added

• Windows binaries of the L3A and L3B processors

Fixed

• Upgrades from previous system installations when the first installed version was older than 1.8.3
• Corrections for SciHub downloads when sites are defined near equator or prime meridian
• Corrections for USGS queries according to the latest changes in the USGS responses
• Corrections for MAJA “MTD_ALL.xml is not valid according to JPI” issue
• PhenoNDVI processor corrections
• Updates of the L3B processor in order to handle during the scheduling also the unprocessed products since the start of the season, not only the ones from the last week
• Corrections for handling site polygons that have holes
• Update of the insert_l2a_product_to_db.py in order to support also L2A MAJA format

Known issues

• The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
• The SAFE formatting application sometimes outputs unusable (e.g. black) previews
• The SAFE and L2A product previews are not gamma-corrected and can be too dark
• The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
• The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
• The crop type and mask training step sometimes crashes while loading the OpenCV models
• MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
• The product formatting and tile aggregation steps are inefficient
• Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
• The trimming step of the Crop Mask processor still uses a large amount of memory
• The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
• The dashboard previews don’t match their bounds rectangle because of projection mismatch
• The website has display issues on Safari
• When deleting a site, the folders for the L2A products that were not processed by MACCS are not deleted and should be deleted manually. Normally these folders contain no valid product and contain only log and EEF files

System Version 2.0 [15/05/2019]: Public release of the version 2.0 of the system

Added

• MAJA (MACCS-ATCOR Joint Algorithm) implementation for the cloud screening and atmospheric correction of the Sentinel-2 and Landsat 8 data
• New visualization tool
• Data sources enhancement to support local DIAS repositories (CreoDIAS, MUNDI, ONDA)
• Failsafe download from SciHub: if a product cannot be accessed/downloaded after several retries from the main data source, it is downloaded automatically from SciHub
• Tile-oriented L2A processing (as opposed to product-oriented) to allow better parallelization
• Retry mechanism for L2A processing: if the L2A processing fails, it is retried several times
• Email alert for abnormal functioning of services
• L1C zip or tar.gz products supported in L2A processor
• Database enhancement by providing more details for MACCS / MAJA processing (logs and cloud, cloud shadow and snow percentages)

System Version 1.8.3 [20/09/2018]: Public release of the version 1.8.3 of the system

Fixed

• Issues during products download from SciHub or USGS
• The issue with configuring a new logging level in application.properties

Added

• Users management from the system IHM
• Data sources configuration from the system IHM
• Filtering products by tile, season or time interval for the Products and Custom Jobs in IHM
• New flags for LAI monodate products based on INRA algorithm

Known issues

• The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
• The SAFE formatting application sometimes outputs unusable (e.g. black) previews
• The SAFE and L2A product previews are not gamma-corrected and can be too dark
• The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
• The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
• The crop type and mask training step sometimes crashes while loading the OpenCV models
• MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
• The MACCS launcher script does not process different tiles of the same site in parallel
• The product formatting and tile aggregation steps are inefficient
• Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
• The trimming step of the Crop Mask processor still uses a large amount of memory
• The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
• The dashboard previews don’t match their bounds rectangle because of projection mismatch
• The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
• The website has display issues on Safari
• When deleting a site, the folders for the L2A products that were not processed by MACCS are not deleted and should be deleted manually. Normally these folders contain no valid product and but contain only log and EEF files.

System Version 1.8.2 [13/07/2018]: Public release of the version 1.8.2 of the system

Fixed

• Updated GIPP
• Handling of multi-polygons site extent geometry
• Decimal precision in geometry
• Handling of the time lag between the Sci-Hub and AWS acquisition availability (normally, this can be tuned by changing values in datasource table for max_retries and retry_interval).

Added

•  A new configuration key was added in order to force queries from the start of the season instead of the last successfuly download date. The config key (that can be set only for passing again all seasons of the site and not for a nominal functionning) can be set to “config” table as “scheduled.lookup.all_products.enabled = true”. After a complete pass of the season is made, this key should be removed or set to false (and sen2agri-services restared) as it does not functions correctly with the near-realtime downloading mode.
• Support for local repositories that have paths in the format 2018/1/2 (and not 2018/01/02) by setting in the sen2agri-services.properties file the following keys:

   AWSDataSource.Sentinel2.usePadding=false
   AWSDataSource.Landsat8.usePadding=false

Known issues

• The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
• The SAFE formatting application sometimes outputs unusable (e.g. black) previews
• The SAFE and L2A product previews are not gamma-corrected and can be too dark
• The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
• The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
• The crop type and mask training step sometimes crashes while loading the OpenCV models
• MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
• The MACCS launcher script does not process different tiles of the same site in parallel
• The product formatting and tile aggregation steps are inefficient
• Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
• The trimming step of the Crop Mask processor still uses a large amount of memory
• The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
• The dashboard previews don’t match their bounds rectangle because of projection mismatch
• The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
• The website has display issues on Safari
• When deleting a site, the folders for the L2A products that were not processed by MACCS are not deleted and should be deleted manually. Normally these folders contain no valid product and contain only log and EEF files.

System Version 1.8.1 [07/05/2018]: Public release of the version 1.8.1 of the system

Fixed

• LAI Multi-Date processors are now functionning also for products obtained using the new L3B monodate algorithm.

Known issues

Please refer to the known issues of the v1.8.

System Version 1.8 [06-04-2018]: Public pre-release of the version 1.8 of the system

Added

• The L4A/L4B processors use a platform-independent algorithm for the training/validation split. This is a step towards allowing the processor to work on other platforms and give the same results.
• Added the sen2agri-services application that is performing the following:
  • Replaces the old downloaders. Now there are defined datasources that allow customization of sources for query and for download. For example, for S2, the query can be performed from SciHub but the download can be performed from AWS or from a local archive repository (if products are already downloaded locally)
  • Offers RESTful services for interrogating system for :
    * sites configurations, enable/disable/delete sites, retrieve the site seasons
    * enable/disable downloading for a site, for a certain satellite
    * start or stop the downloads for a site/satellite
    * retrieve or update the configuration of a datasource
    * receive notifications from the system
• Added the possibility to delete a site from the web interface
• Added the possibility to upload insitu and strata data from the web interface
• Added INRA implementation for the LAI. In this moment, it works only for the Sentinel2 products and is disabled by default.
• GUI Products tab has now the possibility of filtering products by site, season, tiles, product type or by specifying an interval
• Option to disable L8 (downloading and processing) for a site from the IHM
• Added the possibility to delete a scheduled job from the Dashboard in IHM
• Added buttons for Pause/Resume/Cancel a job in the System Monitoring page of the IHM
• When the final product is creating, a lock file is marking that the product is not complete yet.
• Added options in the configuration to delete SRE or FRE file or to compress the outputs when MACCS creates the final product.
• Added a new state for the downloader_history in order to know when a product is currently processing by MACCS.
• Display version of the system in IHM website footer

Changed

• Breaking change: the L4A and L4B processors now use a different algorithm for the trainig/validation split. Output files will be different from the ones in the previous versions. The accuracy scores will often be lower because poorly-represented classes will now be present in the validation set. They weren’t taken into account previously.
• The SampleSelection application places all the features in both the training and the validation sets if there are too few of them and one of them would end up empty according to the configured split (e.g. 75% training with only 2 features). This allows taking them into account for validation, although with lower accuracy.
• Removed L2A processor from the GUI Dashboard tab since it can not be launch from here
• Changed processor names in the GUI Dashboard.
• The downloads stops now when a site is disabled.
• Corrected the LAI to avoid saturation of values.
• Usage of CCI 2015 instead of CCI 2010.

Fixed

• Fixed crash in the SampleSelection application used by the L4 processors when a feature with no geometry is present
• The SampleSelection application always puts features in one of the training or validation set. Previously, they were sometimes lost.
• Composite Python Processor was updated to generate both 10 and 20 m resolutions.
• Corrections in LAI and Pheno NDVI processors Python scripts.

Known issues

• The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
• The SAFE formatting application sometimes outputs unusable (e.g. black) previews
• The SAFE and L2A product previews are not gamma-corrected and can be too dark
• The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
• The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
• The crop type and mask training step sometimes crashes while loading the OpenCV models
• MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
• The MACCS launcher script does not process different tiles of the same site in parallel
• The product formatting and tile aggregation steps are inefficient
• Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
• The trimming step of the Crop Mask processor still uses a large amount of memory
• The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
• The dashboard previews don’t match their bounds rectangle because of projection mismatch
• The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
• The website has display issues on Safari

System Version 1.7 [30-11-2017]: Second public version of the system

### Added
– The system is now able to download and process Sentinel-2B images.
– The unsupervised crop mask processor can now use (optionally, on by default) the red edge bands
– Added the local mode in the Sentinel-2 downloader in order to use a local L1C repositories as products store
– Added a new flag for L1C products processing in order to determine if the product was handled OK or not by MACCS.

### Changed
– Products in the `RT` and `T2` Landsat 8 collections are no longer downloaded
– Changed the crop mask post-filtering to ignore negative reflectance values
– Reduced the CPU usage of the crop mask post-filtering NDVI extraction step by 20%, and the wall clock time by 15% when running with 4 threads
– Reduced the CPU and disk usage by merging the crop mask post-filtering no data filling and PCA steps; CPU usage went down by 27% and disk usage by about 20 GB for a single tile
– The Landsat 8 downloader no longer keep a download status line if it’s not writing to a TTY

### Fixed
– Fixed an issue that made the crop mask post-filtering output invalid rasters when both Sentinel-2 and Landsat 8 products were used for post-filtering; this was disabled by default
– Changed the post-filtered crop mask data type from `Float32` to `Int16`
– Fixed an issue that caused the supervised crop mask processor to crash on tiles spanned by multiple strata
– Fixed an issue that sometimes caused MACCS to be passed an L3C product instead of the previous L2A one
– Fixed a potential division by zero in the Landsat 8 downloader

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step flags as invalid products with even a single tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The MACCS launcher script does not process different tiles of the same site in parallel
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

System version 1.6.1 [05-07-2017]: First public release of the system

### Added
– Landsat 8 collections support for the downloader

### Fixed
– The Landsat 8 downloader now properly reports USGS authentication errors when using a proxy
– Fixed an issue in the retrieval of season dates for sites with more than one season

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step eagerly flags products with a tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari

System version 1.6 [08-06-2017]: Pre-release of the system to Champion-users

### Added
– The possibility to select the input bands when creating the LAI monodate products, including 20M bands and possibility to use or not NDVI and RVI.
– Scripts for deleting a site and for filtering a site tiles
– Multiple seasons in the UI
– The website monitoring page can now display the command line of a job and its output
– The supervised crop mask processor can now use (optionally, on by default) features derived from the red edge bands

### Changed
– In the LAI scheduled jobs now are taken into account the insertion dates instead of product creation date.
– Limited the crop type processor training to `10 000` samples / tile
– The crop type and supervised crop mask processors now include the in-situ data in the output product
– The short name and extent of sites can no longer be updated from the website
– The website monitoring page now sorts jobs by submission time instead of end time
– The installer now sets `executor.listen-ip` to `127.0.0.1` and the SLURM node name to `localhost`
– The binaries were recompiled for `GEOS 1.5`
– Increased crop mask and crop type tile thread hint from `2` to `5` to reduce the risk of out-of-memory conditions

### Fixed
– Changed the advanced mode for processors parameters in the sen2agri-config configuration.
– In IPP file name for L3A is not inserted the used bands_mapping and not the one from the DB.
– The missing L8 HDR from the L3C/L3D IPP file
– Corrected the date order in the creation of L3C/L3D products.
– Fixed a crash in the crop type processor when one of the tiles was missing Landsat 8 data
– Website custom L4A jobs use a value of `40 000` training samples / tile instead of `4 000` to match the processor default

### Known issues
– The training/validation polygon splitting step of the Crop Type and supervised Crop Mask processors can lose polygons
– The SAFE formatting application sometimes outputs mosaics with black edges around tile edges
– The SAFE formatting application sometimes outputs unusable (e.g. black) previews
– The SAFE and L2A product previews are not gamma-corrected and can be too dark
– The SAFE validation step eagerly flags products with a tile having a low-variance band as invalid (`NOTV`)
– The crop type and crop mask processors don’t perform the normalization step properly when using SVM classification
– The crop type and mask training step sometimes crashes while loading the OpenCV models
– MACCS can sometimes crash or hang under high load: https://github.com/InsightSoftwareConsortium/ITK/commit/d68c1c0f23fea97ab782c185095b4ef3259cec68
– The product formatting and tile aggregation steps are inefficient
– Performance of the multi-tile Crop Type and Crop Mask processors can be poor for tiles with a large number of input products, especially on hardware with a large number of cores
– The trimming step of the Crop Mask processor still uses a large amount of memory
– The unsupervised Crop Mask processor expects a reference map with the ESA-CCI LC map labels
– The dashboard previews don’t match their bounds rectangle because of projection mismatch
– The LAI model is created for each tile. The SDD and ATBD should be updated if another behaviour is desired and needs to be implemented.
– The website has display issues on Safari