This is a tutorial about Remotior Sensus, a Python package that allows for the processing of remote sensing images and GIS data.

In particular, this tutorial describes the use of Remotior Sensus in Copernicus JupyterLab, which is a Jupyter Notebook service in a web-based environment, offering several tools for working with the Copernicus Data Space.

This service can be accessed at this link https://jupyterhub.dataspace.copernicus.eu after a free registration to the Copernicus Data Space Ecosystem (CDSE).

The Jupyter Notebooks are available in 3 flavors: Small (2 CPU cores and 4GB RAM), Medium (2 CPU cores and 8GB RAM) and Large (4 CPU cores and 16GB RAM). As stated in the documentation, to ensure the fair use of resources by the CDSE users, it is recommended to start with the Small flavor and switch to a bigger only in case of issues with kernel crashing due to the lack of available memory.

Therefore, the Copernicus JupyterLab offers a great opportunity to use Copernicus data in a cloud environment. In this tutorial, we are going to see how to:

Download and preprocess Sentinel-2 images.
Create a BandSet and prepare a training input
Run a Random Forest classification

All the above steps are performed in the cloud. The classification output is saved in a persistent storage with 10GB of space and can be downloaded later.

You can download the Jupyter Notebook from this link ,

To use this Notebook, after logging in the Copernicus JupyterLab, click the button to upload a file. After uploading this Notebook file, open it and run using the "Geo science" kernel.

Following the output of the Notebook as illustration. Of course you are invited to run the Notebook directly.

Introduction

This notebook illustrates how to perform Random Forest classification of a Sentinel-2 image using Remotior Sensus .

Remotior Sensus (which is Latin for “a more remote sense”) is a Python package that allows for the processing of remote sensing images and GIS data. Other tutorials are available here.

Install Remotior Sensus and start a new Session

First, we install Remotior Sensus using Pip.

In [1]:

!pip install -U remotior_sensus

Requirement already satisfied: remotior_sensus in /opt/conda/envs/geo/lib/python3.11/site-packages (0.1.23)

We import Remotior Sensus and start a Remotior Sensus’ session, defining the number of processes (n_processes) and RAM (available_ram) available in the system.

For further information about sessions, please read the User Manual.

In [2]:

import remotior_sensus
rs = remotior_sensus.Session(n_processes=2, available_ram=3640)

INFO[2023-12-03T14:02:49.822] system_tools.get_system_info[38] version:0.1.23; system: Linux; 64bit: True; n_processes: 2; ram: 3640; temp.dir: /tmp/remotior_sensus_1203ssjy_rv7
WARNING[2023-12-03T14:02:49.822] session._check_dependencies[457] No module named 'torch'

⚠ warning: dependency error: pytorch

Prepare Input Data

For the classification we need to:

Create a BandSet using an image
Load a training input
Perform the random forest classification

A BandSet is an object that includes information about bands (from the file path to the spatial and spectral characteristics).

We can define a working directory where data of this tutorial are saved.

In [3]:

working_directory = 'mystorage'

Before creating the BandSet we should prepare input data. For the purpose of this tutorial, in order to reduce computation time, we are going to download a Copernicus Sentinel-2 image which includes the following bands.

Sentinel-2 Bands	Central Wavelength [micrometers]	Resolution [meters]
Band 2 - Blue	0.490	10
Band 3 - Green	0.560	10
Band 4 - Red	0.665	10
Band 5 - Vegetation Red Edge	0.705	20
Band 6 - Vegetation Red Edge	0.740	20
Band 7 - Vegetation Red Edge	0.783	20
Band 8 - NIR	0.842	10
Band 8A - Vegetation Red Edge	0.865	20
Band 11 - SWIR	1.610	20
Band 12 - SWIR	2.190	20

Image Search

For downloading Copernicus images we need to enter user and passwor of the https://dataspace.copernicus.eu.

In [4]:

from getpass import getpass

user = getpass('Copernicus user: ')
password = getpass('Copernicus password: ')

We use the function download_products.search to query the Sentinel-2 database and retrieve a table of available images:

name_filter allows for searching images that match a tile name
date_from and date_to set a period range
max_cloud_cover sets a threshold for maximum cloud cover
result_number returns a limited number of results, in this case 1 image

For more information about how to download data using Remotior Sensus please read this tutorial .

In [5]:

query_result = rs.download_products.search(
    product='Sentinel-2', name_filter='32TPR',
    date_from='2023-07-01', date_to='2023-07-30',
    max_cloud_cover=10, result_number=1,
    copernicus_user=user, copernicus_password=password
)

INFO[2023-12-03T14:03:05.247] download_products.query_sentinel_2_database[114] start
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): identity.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://identity.dataspace.copernicus.eu:443 "POST /auth/realms/CDSE/protocol/openid-connect/token HTTP/1.1" 200 3354

 search [ 90%]:search in progress ⚙

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(MTD_MSIL2A.xml)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(MTD_MSIL2A.xml)/$value HTTP/1.1" 200 54928
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(MTD_MSIL2A.xml)/$value HTTP/1.1" 200 54928
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): identity.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://identity.dataspace.copernicus.eu:443 "DELETE /auth/realms/CDSE/account/sessions/d92cbd4d-e81a-4f98-bf11-365191a03fcf HTTP/1.1" 204 0
INFO[2023-12-03T14:03:09.467] download_products.query_sentinel_2_database[405] end

 search [100%]: ⬤

The query_result is an object OutputManager including a product table (NumPy recarray) that we can retrive with the attribute extra['product_table We can have a look at the table fields, which include information such as acquisition date, cloud cover, and a preview link. For instance, we can read the records of the first image in the table.

In [6]:

product_table = query_result.extra['product_table']
print(print(product_table[0].pprint()))

         product: Sentinel-2
           image: L2A_T32TPR_A033081_20230707T101428
      product_id: S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302
acquisition_date: 2023-07-07
     cloud_cover: 3
       zone_path: 32TPR
             row: None
         min_lat: 45.034190140253
         min_lon: 10.2700756717775
         max_lat: 46.0462654563034
         max_lon: 11.7105441475977
      collection: None
            size: 1
         preview: https://storage.googleapis.com/gcp-public-data-sentinel-2/L2/tiles/32/T/PR/S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE/GRANULE/L2A_T32TPR_A033081_20230707T101428/QI_DATA/T32TPR_20230707T100559_PVI.jp2
             uid: 302382f3-fb33-4a3a-a5db-5836451b374e
None

To download the images, we use the function download_products.download and download the 10m and 20m bands:

product_table sets the images to be downloaded from the previously query results (in this case one image)
band_list defines a list of band names which for Sentinel-2 are: '02', '03', '04', '05', '06', '07', '08', '8a', '11', '12'.
output_path defines where downloaded files are saved.

During the download, a directory inside the output_directory is created for the downloaded image.

downloaded_image is an an object OutputManager that includes a list of created directories in the attribute extra['directory_paths'].

In [7]:

downloaded_image = rs.download_products.download(
    product_table=product_table,
    band_list=['02', '03', '04', '05', '06', '07', '08', '8a', '11', '12'],
    output_path=working_directory + '/downloaded_images',
    copernicus_user=user, copernicus_password=password
    )

print(downloaded_image.extra['directory_paths'])

INFO[2023-12-03T14:03:09.504] download_products.download[518] start
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): identity.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://identity.dataspace.copernicus.eu:443 "POST /auth/realms/CDSE/protocol/openid-connect/token HTTP/1.1" 200 3354
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(MTD_MSIL2A.xml)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443

 download products [  0%]:starting ⚙

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(MTD_MSIL2A.xml)/$value HTTP/1.1" 200 54928
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(MTD_MSIL2A.xml)/$value HTTP/1.1" 200 54928
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B02_10m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B02_10m.jp2)/$value HTTP/1.1" 200 119714873

 download products [  0%]:starting ⚙

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B02_10m.jp2)/$value HTTP/1.1" 200 119714873

 download products [  8%] [elapsed 0min51sec] [remaining 9min55sec] downloading band 02 ◕K

ERROR[2023-12-03T14:04:02.219] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B02.jp2']

 download products [  8%] [elapsed 0min52sec] [remaining 9min55sec] downloading band 02 ◕

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B03_10m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B03_10m.jp2)/$value HTTP/1.1" 200 123641935

 download products [  8%] [elapsed 0min57sec] [remaining 9min55sec]:downloading band 02 ◕

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B03_10m.jp2)/$value HTTP/1.1" 200 123641935

 download products [ 16%] [elapsed 1min05sec] [remaining 5min45sec] downloading band 03 ◑

ERROR[2023-12-03T14:04:16.233] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B03.jp2']

 download products [ 17%] [elapsed 1min06sec] [remaining 5min26sec] downloading band 03 ◑

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B04_10m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B04_10m.jp2)/$value HTTP/1.1" 200 123801853

 download products [ 17%] [elapsed 1min17sec] [remaining 5min26sec]:downloading band 03 ◑

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B04_10m.jp2)/$value HTTP/1.1" 200 123801853

 download products [ 27%] [elapsed 1min27sec] [remaining 3min55sec] downloading band 04 ◕

ERROR[2023-12-03T14:04:38.248] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B04.jp2']

 download products [ 28%] [elapsed 1min28sec] [remaining 3min48sec] downloading band 04 ◕

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B05_20m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B05_20m.jp2)/$value HTTP/1.1" 200 33830920

 download products [ 28%] [elapsed 1min29sec] [remaining 3min48sec]:downloading band 04 ◕

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B05_20m.jp2)/$value HTTP/1.1" 200 33830920

 download products [ 31%] [elapsed 1min31sec] [remaining 3min24sec]:downloading band 05 ○

ERROR[2023-12-03T14:04:42.267] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B05.jp2']

 download products [ 31%] [elapsed 1min32sec] [remaining 3min24sec]:downloading band 05 ○

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B06_20m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B06_20m.jp2)/$value HTTP/1.1" 200 33738033

 download products [ 31%] [elapsed 1min35sec] [remaining 3min24sec] downloading band 05 ○

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B06_20m.jp2)/$value HTTP/1.1" 200 33738033

 download products [ 42%] [elapsed 1min39sec] [remaining 2min16sec] downloading band 06 ◔

ERROR[2023-12-03T14:04:50.294] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B06.jp2']

 download products [ 43%] [elapsed 1min40sec] [remaining 2min14sec] downloading band 06 ◔

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B07_20m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B07_20m.jp2)/$value HTTP/1.1" 200 33778240

 download products [ 43%] [elapsed 1min45sec] [remaining 2min14sec]:downloading band 06 ◔

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B07_20m.jp2)/$value HTTP/1.1" 200 33778240

 download products [ 58%] [elapsed 2min10sec] [remaining 1min33sec]:downloading band 07 ◕

ERROR[2023-12-03T14:05:21.318] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B07.jp2']

 download products [ 59%] [elapsed 2min11sec] [remaining 1min32sec]:downloading band 07 ◕

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B08_10m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B08_10m.jp2)/$value HTTP/1.1" 200 135422775

 download products [ 59%] [elapsed 2min18sec] [remaining 1min32sec] downloading band 07 ◕

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R10m)/Nodes(T32TPR_20230707T100559_B08_10m.jp2)/$value HTTP/1.1" 200 135422775

 download products [ 65%] [elapsed 2min25sec] [remaining 1min19sec] downloading band 08 ◑

ERROR[2023-12-03T14:05:36.332] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B08.jp2']

 download products [ 66%] [elapsed 2min26sec] [remaining 1min16sec] downloading band 08 ◑

ERROR[2023-12-03T14:05:37.339] multiprocess_manager.multi_download_file[2353] Error proc 0-['.jp2']
ERROR[2023-12-03T14:05:37.340] download_products._check_sentinel_2_bands[991] error: download failed L2A_T32TPR_A033081_20230707_B8a

 download products [ 66%] [elapsed 2min27sec] [remaining 1min16sec]:downloading band 08 ◑▲ error: failed download L2A_T32TPR_A033081_20230707_B8a

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B11_20m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B11_20m.jp2)/$value HTTP/1.1" 200 33738369

 download products [ 66%] [elapsed 2min30sec] [remaining 1min16sec] downloading band 08 ◑

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B11_20m.jp2)/$value HTTP/1.1" 200 33738369

 download products [ 81%] [elapsed 2min33sec] [remaining 0min36sec]:downloading band 11 ○

ERROR[2023-12-03T14:05:44.345] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B11.jp2']

 download products [ 82%] [elapsed 2min34sec] [remaining 0min34sec]:downloading band 11 ○

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): catalogue.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://catalogue.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B12_20m.jp2)/$value HTTP/1.1" 301 162
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): zipper.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B12_20m.jp2)/$value HTTP/1.1" 200 33832340

 download products [ 82%] [elapsed 2min41sec] [remaining 0min34sec] downloading band 11 ○

DEBUG:urllib3.connectionpool:https://zipper.dataspace.copernicus.eu:443 "GET /odata/v1/Products(302382f3-fb33-4a3a-a5db-5836451b374e)/Nodes(S2B_MSIL2A_20230707T100559_N0509_R022_T32TPR_20230707T131302.SAFE)/Nodes(GRANULE)/Nodes(L2A_T32TPR_A033081_20230707T101428)/Nodes(IMG_DATA)/Nodes(R20m)/Nodes(T32TPR_20230707T100559_B12_20m.jp2)/$value HTTP/1.1" 200 33832340

 download products [ 91%] [elapsed 2min44sec] [remaining 0min16sec]:downloading band 12 ○

ERROR[2023-12-03T14:05:55.372] multiprocess_manager.multi_download_file[2353] Error proc 0-['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07/T32TPR_A033081_20230707T101428_B12.jp2']
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): identity.dataspace.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://identity.dataspace.copernicus.eu:443 "DELETE /auth/realms/CDSE/account/sessions/ed4ab937-27d8-4d7c-a84c-ff60b648d12e HTTP/1.1" 204 0
INFO[2023-12-03T14:05:55.445] download_products.download[833] end

 download products [100%] [elapsed 2min45sec]: ⬤['mystorage/downloaded_images/L2A_T32TPR_A033081_20230707T101428_2023-07-07']

Data Preprocessing

To preprocess the images (converte from digital numbers to decimal values of reflectance) we use the function preprocess_products.preprocess:

sensor sets the sensor of input image, in this case 'Sentinel-2'
input_path accepts a directory containing the image bands, which in this case we get from downloaded_image.extra['directory_paths'][0]
output_path sets output directory where converted bands are saved, in this case output_path=working_directory + '/preprocessed'
nodata_value allows for masking image NoData by value.

Defining a list preprocessed_image_directories is convenient to retrieve the directories of preprocessed images that we are going to use later. The process can take a few minutes.

In [8]:

preprocessed = rs.preprocess_products.preprocess(
    sensor='Sentinel-2',
    input_path=downloaded_image.extra['directory_paths'][0],
    output_path=working_directory + '/preprocessed',
    nodata_value=0
    )

INFO[2023-12-03T14:05:55.538] preprocess_products.perform_preprocess[137] start

 preprocess products [  0%]:starting ⚙

/opt/conda/envs/geo/lib/python3.11/site-packages/remotior_sensus/core/processor.py:536: RuntimeWarning: invalid value encountered in cast
  value_as_nodata = np.asarray(value_as_nodata).astype(

 preprocess products [  0%]:starting ⚙

/opt/conda/envs/geo/lib/python3.11/site-packages/remotior_sensus/core/processor.py:536: RuntimeWarning: invalid value encountered in cast
  value_as_nodata = np.asarray(value_as_nodata).astype(

 preprocess products [ 89%] [elapsed 2min54sec] [remaining 0min21sec] processing ⬤

INFO[2023-12-03T14:08:50.932] preprocess_products.perform_preprocess[628] end; preprocess products: ['mystorage/preprocessed/T32TPR_A033081_20230707T101428_B02.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B03.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B04.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B05.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B06.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B07.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B08.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B11.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B12.tif']

 preprocess products [100%] [elapsed 2min55sec]: ⬤ining 0min21sec]:processing ⚙

1 Create the Input Bandset

Now, we create a BandSet Catalog which contains BandSets (note that an empty BandSet 1 will be automatically created).

In [9]:

catalog = rs.bandset_catalog()

INFO[2023-12-03T14:08:50.936] bandset_catalog.create[744] start
INFO[2023-12-03T14:08:50.937] bandset_catalog.create[751] end

Now we create a BandSet using catalog.create_bandset:

paths defines the list of paths to the bands, in this case the directory of preprocessed files [working_directory + '/preprocessed'] can be used to get all the bands inside this directory
bandset_number explicitly sets the number of the BandSet as 1
wavelengths sets a listo of values as the center wavelength for each band, in this case ['Sentinel-2'] automatically defines Sentinel-2 values for the identified bands in the BandSet

Optionally, we can set the box_coordinate_list argument as list of coordinates [left, top, right, bottom] to create a virtual subset and reduce computation time for a smaller area of interest.

In [10]:

catalog.create_bandset(
    paths=[working_directory + '/preprocessed'],
    bandset_number=1, wavelengths=['Sentinel-2'],
    box_coordinate_list=[629000, 5036000, 641000,5028000],
    )

INFO[2023-12-03T14:08:50.943] bandset_catalog.create[744] start
INFO[2023-12-03T14:08:51.930] bandset_catalog.create[948] end; file list: ['mystorage/preprocessed/T32TPR_A033081_20230707T101428_B02.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B03.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B04.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B05.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B06.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B07.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B08.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B11.tif', 'mystorage/preprocessed/T32TPR_A033081_20230707T101428_B12.tif']

Out[10]:

<remotior_sensus.core.bandset_catalog.BandSet at 0x7f9cb522d410>

We get the first BandSet in the catalog with the function catalog.get(1), then we can print the created BandSet with the BandSet function print(). This is the input that is going to be classified.

In [11]:

catalog.get(1).print()

name: T32TPR_A033081_20230707T101428_B 
date: NaT 
root directory: None 
crs: PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] 
box coordinate left │  629000 
box coordinate top │  5036000 
box coordinate right │  641000 
box coordinate bottom │  5028000 
┌─────────────┬─────────────┬───────────────────────────────────────────────────────────────┬────────────────┬────────────────────────────────────┬────────────┬─────────────────┬─────────────────┬───────────────────────┬──────┬────────┬────────┬───────────┬──────────┬───────────┬──────────┬─────────┬─────────┬────────┬───────────┬─────────────────┬──────────────┬──────────────┬────────┬────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ 
│ band_number │ raster_band │ path                                                          │ root_directory │ name                               │ wavelength │ wavelength_unit │ additive_factor │ multiplicative_factor │ date │ x_size │ y_size │ top       │ left     │ bottom    │ right    │ x_count │ y_count │ nodata │ data_type │ number_of_bands │ x_block_size │ y_block_size │ scale  │ offset │ crs                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  │ 
├─────────────┼─────────────┼───────────────────────────────────────────────────────────────┼────────────────┼────────────────────────────────────┼────────────┼─────────────────┼─────────────────┼───────────────────────┼──────┼────────┼────────┼───────────┼──────────┼───────────┼──────────┼─────────┼─────────┼────────┼───────────┼─────────────────┼──────────────┼──────────────┼────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ 
│ 1           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B02.tif │ None           │ T32TPR_A033081_20230707T101428_B02 │ 0.49       │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 10.0   │ 10.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 10980   │ 10980   │ 65535  │ UInt16    │ 1               │ 10980        │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 2           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B03.tif │ None           │ T32TPR_A033081_20230707T101428_B03 │ 0.56       │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 10.0   │ 10.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 10980   │ 10980   │ 65535  │ UInt16    │ 1               │ 10980        │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 3           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B04.tif │ None           │ T32TPR_A033081_20230707T101428_B04 │ 0.665      │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 10.0   │ 10.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 10980   │ 10980   │ 65535  │ UInt16    │ 1               │ 10980        │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 4           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B05.tif │ None           │ T32TPR_A033081_20230707T101428_B05 │ 0.705      │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 20.0   │ 20.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 5490    │ 5490    │ 65535  │ UInt16    │ 1               │ 5490         │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 5           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B06.tif │ None           │ T32TPR_A033081_20230707T101428_B06 │ 0.74       │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 20.0   │ 20.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 5490    │ 5490    │ 65535  │ UInt16    │ 1               │ 5490         │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 6           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B07.tif │ None           │ T32TPR_A033081_20230707T101428_B07 │ 0.783      │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 20.0   │ 20.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 5490    │ 5490    │ 65535  │ UInt16    │ 1               │ 5490         │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 7           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B08.tif │ None           │ T32TPR_A033081_20230707T101428_B08 │ 0.842      │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 10.0   │ 10.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 10980   │ 10980   │ 65535  │ UInt16    │ 1               │ 10980        │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 8           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B11.tif │ None           │ T32TPR_A033081_20230707T101428_B11 │ 1.61       │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 20.0   │ 20.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 5490    │ 5490    │ 65535  │ UInt16    │ 1               │ 5490         │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
│ 9           │ 1           │ mystorage/preprocessed/T32TPR_A033081_20230707T101428_B12.tif │ None           │ T32TPR_A033081_20230707T101428_B12 │ 2.19       │ µm (1 E-6m)     │ 0.0             │ 1.0                   │ NaT  │ 20.0   │ 20.0   │ 5100000.0 │ 600000.0 │ 4990200.0 │ 709800.0 │ 5490    │ 5490    │ 65535  │ UInt16    │ 1               │ 5490         │ 1            │ 0.0001 │ 0.0    │ PROJCS["WGS 84 / UTM zone 32N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",9],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32632"]] │ 
└─────────────┴─────────────┴───────────────────────────────────────────────────────────────┴────────────────┴────────────────────────────────────┴────────────┴─────────────────┴─────────────────┴───────────────────────┴──────┴────────┴────────┴───────────┴──────────┴───────────┴──────────┴─────────┴─────────┴────────┴───────────┴─────────────────┴──────────────┴──────────────┴────────┴────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

We can create a virtual raster of the BandSet which we can open in GIS software.

In [12]:

vrt_file = working_directory + '/virtual.vrt'
catalog.create_virtual_raster(
    bandset_number=1, output_path=vrt_file
    )

Out[12]:

'mystorage/virtual.vrt'

We can display the virtual raster (using matplotlib).

In [13]:

#@title Figure
from osgeo import gdal
import numpy
import matplotlib.colors as mcolors
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt

raster = gdal.Open(vrt_file)
blue_band = raster.GetRasterBand(1).ReadAsArray()
green_band = raster.GetRasterBand(2).ReadAsArray()
red_band = raster.GetRasterBand(3).ReadAsArray()
image = numpy.stack((red_band, green_band, blue_band), axis=-1)
image = numpy.clip((image/10000) / 0.2, 0, 1)
plt.imshow(image)
plt.rcParams['figure.figsize'] = (15, 15)
plt.axis('off')
plt.show()
band = image = plt = None

2 Load the Training Input

In general, we would need to create a vector file containing polygons (ROIs) used to train the algorithm, where each polygon corresponds to a class (or macroclass).

In this tutorial, the following macroclasses are considered.

Macroclass name	Macroclass ID
Water	1
Built-up	2
Vegetation	3
Soil	4

For the creation of these polygons, one can use GIS software such as the Semi-Automatic Classification Plugin for QGIS. For the purpose of this tutorial, we are going to download an already prepared geopackage of polygons.

More information about classes and macroclasses is available here.

In [14]:

import requests
request = requests.get('https://docs.google.com/uc?id=1onFqHdk1aTN9WYb6ZoUnAkZMiZEUoITd')

file = open(working_directory + '/training.gpkg', 'wb')
file.write(request.content)
file.close()

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): docs.google.com:443
DEBUG:urllib3.connectionpool:https://docs.google.com:443 "GET /uc?id=1onFqHdk1aTN9WYb6ZoUnAkZMiZEUoITd HTTP/1.1" 303 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): doc-14-5k-docs.googleusercontent.com:443
DEBUG:urllib3.connectionpool:https://doc-14-5k-docs.googleusercontent.com:443 "GET /docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/cji4lrtbn56a73cbqjk3pm6d1v5kk97s/1701612525000/15733009328155748858/*/1onFqHdk1aTN9WYb6ZoUnAkZMiZEUoITd?uuid=0d3f214b-07bf-48e4-8b6d-3935a5e3f3fb HTTP/1.1" 200 147456

To open the training input, first we create a Spectral Signatures Catalog using SpectralSignaturesCatalog, which must be created specifically for a BandSet (and the characteristics thereof):

bandset sets the reference BandSet

In [15]:

bandset_1 = catalog.get(1)

signature_catalog = rs.spectral_signatures_catalog(
    bandset=bandset_1
    )

Now we can import the geopackage in the Spectral Signatures Catalog using the function import_vector:

file_path sets the path of the input file, in this case working_directory + '/training.gpkg'
macroclass_field sets the name of the field corresponding to the macroclass ID in the vector
macroclass_name_field sets the name of the field corresponding to the macroclass name in the vector
class_field sets the name of the field corresponding to the class ID in the vector
class_name_field sets the name of the field corresponding to the class name in the vector

In [16]:

signature_catalog.import_vector(
    file_path=working_directory + '/training.gpkg',
    macroclass_field='macroclass_id',
    macroclass_name_field='macroclass_name',
    class_field='class_id',
    class_name_field='class_name'
    )

 remotior_sensus [ 36%]:calculate signature ⚙

3 Perform Random Forest Classification

We are going to train the Random Forest classifier and perform the classification.

To classify the image we are going to use the tool band_classification, with the following parameters:

input_bands is a parameter that accepts a list of input raster paths, or a BandSet number, or a previously defined BandSet
bandset_catalog the catalog containing the BandSet (in case input_bands is a BandSet number)
spectral_signatures the SpectralSignaturesCatalog containing the training input
macroclass if True, the training in evaluated on the macroclass ID field; if False, the training is evaluated on the class ID field
output_path sets the output path of the classification raster
save_classifier is an optional parameter, if True saves the classifier to a file to be loaded (without training) for a new classification
classification_confidence is an optional parameter, if True, writes a classification confidence raster as additional output, useful to evaluate the confidence of the classifier at pixel level (from 0=low to 1=high confidence)
algorithm_name is the name of the classification algorithms, which for Random Forest is 'rf', other options are:
- Minimum Distance = 'md'
- Maximum Likelihood = 'ml'
- Spectral Angle Mapping = 'sam'
- Random Forest OneVsRest = 'rf_ovr'
- Support Vector Machine = 'svm'
- Multi Layer Perceptron = 'mlp'
- Multi Layer Perceptron using PyTorch framework = 'pytorch_mlp'
  classification_confidence=True Available parameters for Random Forest are:
rf_max_features parameter that sets the number of features considered in node splitting; if None all features are considered in node splitting, available options are ‘sqrt’ as square root of all the features, an integer number, or a float number for a fraction of all the features.
rf_number_trees parameter that sets the number of trees in the forest
rf_min_samples_split parameter that sets the minimum number of samples required to split an internal node

For the purpose of this tutorial we set rf_number_trees=50 to reduce computation time, but generally it should be at least rf_number_trees=200 to get good results.

In [17]:

classification = rs.band_classification(
    input_bands=1,
    bandset_catalog=catalog,
    spectral_signatures=signature_catalog,
    macroclass=True,
    output_path=working_directory + '/classification_rf.tif',
    classification_confidence=True,
    algorithm_name='rf',
    rf_number_trees=50,
    rf_min_samples_split=2
    )

INFO[2023-12-03T14:09:53.001] band_classification.band_classification[1255] start

 band classification [  2%] [elapsed 0min06sec] [remaining 4min16sec]:get band arrays ◑ier ⚙

INFO[2023-12-03T14:10:00.401] band_classification.train[450] start

 band classification [  4%] [elapsed 0min10sec] [remaining 2min53sec] get band arrays ⚙

[Parallel(n_jobs=1)]: Done  40 tasks      | elapsed:    3.3s

 band classification [  4%] [elapsed 0min14sec] [remaining 2min53sec] get band arrays ⚙

[Parallel(n_jobs=1)]: Done  40 tasks      | elapsed:    3.3s

 band classification [ 20%] [elapsed 0min20sec] [remaining 1min06sec] fitting ⬤building tree 1 of 50
building tree 2 of 50
building tree 3 of 50
building tree 4 of 50

[Parallel(n_jobs=2)]: Using backend ThreadingBackend with 2 concurrent workers.

building tree 5 of 50
building tree 6 of 50
building tree 7 of 50
building tree 8 of 50
building tree 9 of 50
building tree 10 of 50
building tree 11 of 50
building tree 12 of 50
building tree 13 of 50
building tree 14 of 50
building tree 15 of 50
building tree 16 of 50
building tree 17 of 50
building tree 18 of 50
building tree 19 of 50
building tree 20 of 50
building tree 21 of 50
building tree 22 of 50
building tree 23 of 50
building tree 24 of 50
building tree 25 of 50
building tree 26 of 50
building tree 27 of 50
building tree 28 of 50
building tree 29 of 50
building tree 30 of 50
building tree 31 of 50
building tree 32 of 50
building tree 33 of 50
building tree 34 of 50
building tree 35 of 50
building tree 36 of 50
building tree 37 of 50
building tree 38 of 50
building tree 39 of 50
building tree 40 of 50
building tree 41 of 50
building tree 42 of 50

[Parallel(n_jobs=2)]: Done  37 tasks      | elapsed:    2.0s

building tree 43 of 50
building tree 44 of 50
building tree 45 of 50
building tree 46 of 50
building tree 47 of 50
building tree 48 of 50
building tree 49 of 50
building tree 50 of 50

[Parallel(n_jobs=2)]: Done  50 out of  50 | elapsed:    2.6s finished
INFO[2023-12-03T14:10:16.365] band_classification.train[589] feature importance: [0.17973156 0.0043312  0.01734463 0.00705488 0.00801197 0.00976016
 0.03111353 0.73486082 0.00779125]; accuracy score: 0.9870499052432091
INFO[2023-12-03T14:10:16.366] band_classification.train[1007] end

 band classification [ 20%] [elapsed 0min28sec] [remaining 1min06sec]:fitting ⬤

[Parallel(n_jobs=1)]: Done  40 tasks      | elapsed:    2.0s

 band classification [ 20%] [elapsed 0min29sec] [remaining 1min06sec] fitting ⬤

[Parallel(n_jobs=1)]: Done  40 tasks      | elapsed:    2.1s

 band classification [ 81%] [elapsed 0min31sec] [remaining 0min07sec] writing raster ⚙

INFO[2023-12-03T14:10:25.545] band_classification.band_classification[1358] end; prediction: mystorage/classification_rf.tif

 band classification [ 81%] [elapsed 0min32sec] [remaining 0min07sec] writing raster ⚙

We can display the resulting classification raster.

In [18]:

#@title Figure
from osgeo import gdal
import matplotlib.colors as mcolors
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt

image = gdal.Open(classification.path)
band = image.GetRasterBand(1).ReadAsArray()
color_map = mcolors.ListedColormap(['blue', 'red', 'green', 'yellow'])
plt.imshow(band, cmap=color_map, vmin=1, vmax=4)
plt.rcParams['figure.figsize'] = (15, 15)
plt.axis('off')
legend_handles = [mpatches.Patch(color=c, label=str(val)) for val, c in zip(['water', 'built-up', 'vegetation', 'soil'], ['blue', 'red', 'green', 'yellow'])]
plt.legend(handles=legend_handles, title='Classes', bbox_to_anchor=(0, 1), loc='upper left')
plt.show()
band = image = plt = None

We can also display the confidence raster of the classification, which is a raster having values ranging from 0 (low confidence) to 1 (high confidence).

We can use this confidence raster to assess the quality of the classification and collect additional training areas to improve the results.

In [19]:

#@title Figure
from osgeo import gdal
import numpy
import matplotlib.colors as mcolors
import matplotlib.pyplot as plt

image = gdal.Open(classification.extra['algorithm_raster'])
band = image.GetRasterBand(1).ReadAsArray()
plt.imshow(band, cmap='viridis', vmin=0, vmax=1)
plt.rcParams['figure.figsize'] = (15, 15)
plt.axis('off')

color_values = numpy.linspace(0, 1, 5)
color_map = plt.cm.viridis
legend_handles = []
for v in color_values:
    legend_handles.append(mpatches.Patch(color=color_map(v), label=str(v)))
plt.legend(handles=legend_handles, title='Confidence', bbox_to_anchor=(0, 1), loc='upper left')

plt.show()
band = image = plt = None

Close the Session

The session should be closed at the end of all the processes to remove the temporary files and stop subprocesses.

In [20]:

rs.close()

For any comment or question, join the Facebook group or GitHub discussions about the Semi-Automatic Classification Plugin.

Tutorial: Using Remotior Sensus in Copernicus JupyterLab