EarthCARE ESA MAAP Data Access Example

EarthCARE ESA MAAP Data Access Example#

@ ESA, 2025 = Licensed under “European Space Agency Community License”

Author: Saskia Brose (saskia.brose@esa.int)

Last updated: 05-11-2025

NEW features:

Long lasting token
JSON keys for EarthCARE data files renamed

This script shows how to query the ESA MAAP catalog, stream/download EarthCARE data and then plot the cloud water path and cloud top tempetaure directly from the streamed file.

Prerequisities#

from pystac_client import Client
import fsspec
import xarray as xr
import matplotlib.pyplot as plt
from tqdm import tqdm
import pandas as pd 
import requests
from IPython.display import Image, display
import pathlib

Using the STAC API to query the ESA MAAP stac catalog#

While the discovery of data (querying the ESA MAAP catalogue) does not require any authentication or authorization, accessing the data requires a token generated with an authorized ESA account (EO Sign in) to verify the user. This is the same account and credentials you will have used for the OADS system.

catalog_url = 'https://catalog.maap.eo.esa.int/catalogue/'
catalog = Client.open(catalog_url)

The EarthCARE collections have the same name as previously on OADS, but have the extension _MAAP to distinguish them from the OADS collections. Currently the latest two baselines are provided and there are 12 collections:

Open and Free	Restricted to special users	Restricted CalVal users
EarthCAREL1InstChecked_MAAP	EarthCAREL0L1Products_MAAP	EarthCAREL1Validated_MAAP
EarthCAREL2InstChecked_MAAP	EarthCAREL2Products_MAAP	EarthCAREL2Validated_MAAP
JAXAL2InstChecked_MAAP	EarthCAREAuxiliary_MAAP	JAXAL2Validated_MAAP
EarthCAREOrbitData_MAAP	JAXAL2Products_MAAP
EarthCAREXMETL1DProducts10_MAAP

The first step is to select a collection. You can select multiple collections by extending the list. Note that if you pass multiple collections, the query filters you use must exist across both collections!

# Select one or more collection(s)
EC_COLLECTION = ['EarthCAREL2Validated_MAAP']
EC_multiple_COLLECTIONS = ['EarthCAREL2Validated_MAAP', 'EarthCAREL1Validated_MAAP']

The second step is to further narrow down your search:

Datetime represents the temporal coverage of the data. None can be used for both start and end to indicated unbounded queries.

bbox is defined by the bottom left corner (longmin latmin) and the top right corner coordinates (longmax latmax). The default is [-180,-90,180,90].

Filter – allows you to search based on different metadata parameters.
To understand which queryables exist, you can visit:
https://catalog.maap.eo.esa.int/catalogue/collections/NAMEOFCOLLECTION/queryables

Multiple filters can be combined with or and and following a boolean type of logic.

Examples include:

productType
frame
processingLevel
instrument
orbitNumber

search = catalog.search(
    collections=EC_COLLECTION, 
    filter="productType = 'MSI_COP_2A' and productVersion = 'ba'", # For example filter by product type and baseline. Use boolean logic for multi-filter queries
    bbox = [0, -20, 10, -10],
    datetime = ['2025-10-01T00:00:00Z', '2025-10-22T23:59:59Z'], # Filter by date
    method = 'GET', # This is necessary 
    max_items=5  # Adjust as needed, given the large amount of products it is recommended to set a limit if especially if you display results in pandas dataframe or similiar
)
items = list(search.items()) # Get all items as a list
#items = search.item_collection() # Get all items as a STAC ItemCollection
results = search.matched()

print(f"{results} items found that matched the query.")
print(f"Accessing {len(items)} items (limited by max_items).")

26 items found that matched the query.
Accessing 5 items (limited by max_items).

# Inspect results 
for item in items:
    print(f"Item ID: {item.id}")
    print(f"  - Datetime: {item.datetime}")
    print(f"  - BBOX: {item.bbox}")
    print(f"  - Properties: {item.properties}")
    print(f"  - Assets: {list(item.assets.keys())}")
    print()

Item ID: ECA_EXBA_MSI_COP_2A_20251004T014358Z_20251004T023139Z_07673A
  - Datetime: 2025-10-04 01:43:58+00:00
  - BBOX: [-3.943258, -22.733236, 6.340663, 22.57464]
  - Properties: {'start_datetime': '2025-10-04T01:43:58.000Z', 'end_datetime': '2025-10-04T01:55:34.000Z', 'processing:facility': 'PDGS-CPFxx', 'product:type': 'MSI_COP_2A', 'sat:anx_datetime': '2025-10-04T00:17:13.853Z', 'title': 'ECA_EXBA_MSI_COP_2A_20251004T014358Z_20251004T023139Z_07673A', 'platform': 'EarthCARE', 'datetime': '2025-10-04T01:43:58.000Z', 'instruments': ['MSI'], 'constellation': 'EarthCARE', 'sat:orbit_state': 'ascending', 'processing:software': {'M-CLD': '11.30'}, 'eof-eos:mission_phase': 'Routine', 'grid:code': 'WRS-7673-A', 'processing:level': 'L2A', 'sci:doi': '10.57780/eca-94b4e53', 'created': '2025-10-04T02:31:39.000Z', 'published': '2025-10-13T11:36:52.030Z', 'version': 'BA', 'auth:schemes': {'s3': {'type': 's3'}, 'oidc': {'openIdConnectUrl': 'https://iam.ascend.icsgate.eu/realms/esa-maap/.well-known/openid-configuration', 'type': 'openIdConnect'}}, 'processing:datetime': '2025-10-04T02:31:39.000Z', 'eopf:instrument_mode': 'DEFAULT', 'updated': '2025-10-13T17:21:31Z', 'sat:absolute_orbit': 7673, 'product:acquisition_type': 'other'}
  - Assets: ['thumbnail', 'product', 'metadata_ogc_10_157r4', 'enclosure_h5', 'metadata_ogc_17_003r2', 'metadata_iso_19139', 'enclosure_hdr', 'quicklook']

Item ID: ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E
  - Datetime: 2025-10-13 13:14:44+00:00
  - BBOX: [4.13775, -22.718754, 14.422692, 22.589455]
  - Properties: {'start_datetime': '2025-10-13T13:14:44.000Z', 'end_datetime': '2025-10-13T13:26:20.000Z', 'processing:facility': 'PDGS-CPFxx', 'product:type': 'MSI_COP_2A', 'sat:anx_datetime': '2025-10-13T12:34:19.778Z', 'title': 'ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E', 'platform': 'EarthCARE', 'datetime': '2025-10-13T13:14:44.000Z', 'instruments': ['MSI'], 'constellation': 'EarthCARE', 'sat:orbit_state': 'descending', 'processing:software': {'M-CLD': '11.30'}, 'eof-eos:mission_phase': 'Routine', 'grid:code': 'WRS-7820-E', 'processing:level': 'L2A', 'sci:doi': '10.57780/eca-94b4e53', 'created': '2025-10-13T16:24:07.000Z', 'published': '2025-10-17T11:36:12.056Z', 'version': 'BA', 'auth:schemes': {'s3': {'type': 's3'}, 'oidc': {'openIdConnectUrl': 'https://iam.ascend.icsgate.eu/realms/esa-maap/.well-known/openid-configuration', 'type': 'openIdConnect'}}, 'processing:datetime': '2025-10-13T16:24:07.000Z', 'eopf:instrument_mode': 'DEFAULT', 'updated': '2025-10-17T13:51:49Z', 'sat:absolute_orbit': 7820, 'product:acquisition_type': 'other'}
  - Assets: ['thumbnail', 'product', 'metadata_ogc_10_157r4', 'enclosure_h5', 'metadata_ogc_17_003r2', 'metadata_iso_19139', 'enclosure_hdr', 'quicklook']

Item ID: ECA_EXBA_MSI_COP_2A_20251020T132215Z_20251020T162940Z_07929E
  - Datetime: 2025-10-20 13:22:15+00:00
  - BBOX: [2.265359, -22.686682, 12.536523, 22.561153]
  - Properties: {'start_datetime': '2025-10-20T13:22:15.000Z', 'end_datetime': '2025-10-20T13:33:50.000Z', 'processing:facility': 'PDGS-CPFxx', 'product:type': 'MSI_COP_2A', 'sat:anx_datetime': '2025-10-20T12:41:50.623Z', 'title': 'ECA_EXBA_MSI_COP_2A_20251020T132215Z_20251020T162940Z_07929E', 'platform': 'EarthCARE', 'datetime': '2025-10-20T13:22:15.000Z', 'instruments': ['MSI'], 'constellation': 'EarthCARE', 'sat:orbit_state': 'descending', 'processing:software': {'M-CLD': '11.30'}, 'eof-eos:mission_phase': 'Routine', 'grid:code': 'WRS-7929-E', 'processing:level': 'L2A', 'sci:doi': '10.57780/eca-94b4e53', 'created': '2025-10-20T16:29:40.000Z', 'published': '2025-10-20T17:34:18.048Z', 'version': 'BA', 'auth:schemes': {'s3': {'type': 's3'}, 'oidc': {'openIdConnectUrl': 'https://iam.ascend.icsgate.eu/realms/esa-maap/.well-known/openid-configuration', 'type': 'openIdConnect'}}, 'processing:datetime': '2025-10-20T16:29:40.000Z', 'eopf:instrument_mode': 'DEFAULT', 'updated': '2025-10-20T19:32:35Z', 'sat:absolute_orbit': 7929, 'product:acquisition_type': 'other'}
  - Assets: ['thumbnail', 'product', 'metadata_ogc_10_157r4', 'enclosure_h5', 'metadata_ogc_17_003r2', 'metadata_iso_19139', 'enclosure_hdr', 'quicklook']

Item ID: ECA_EXBA_MSI_COP_2A_20251002T015505Z_20251002T023847Z_07642A
  - Datetime: 2025-10-02 01:55:05+00:00
  - BBOX: [-6.714505, -22.686855, 3.556215, 22.553251]
  - Properties: {'start_datetime': '2025-10-02T01:55:05.000Z', 'end_datetime': '2025-10-02T02:06:40.000Z', 'processing:facility': 'PDGS-CPFxx', 'product:type': 'MSI_COP_2A', 'sat:anx_datetime': '2025-10-02T00:28:19.754Z', 'title': 'ECA_EXBA_MSI_COP_2A_20251002T015505Z_20251002T023847Z_07642A', 'platform': 'EarthCARE', 'datetime': '2025-10-02T01:55:05.000Z', 'instruments': ['MSI'], 'constellation': 'EarthCARE', 'sat:orbit_state': 'ascending', 'processing:software': {'M-CLD': '11.30'}, 'eof-eos:mission_phase': 'Routine', 'grid:code': 'WRS-7642-A', 'processing:level': 'L2A', 'sci:doi': '10.57780/eca-94b4e53', 'created': '2025-10-02T02:38:47.000Z', 'published': '2025-10-14T05:22:37.274Z', 'version': 'BA', 'auth:schemes': {'s3': {'type': 's3'}, 'oidc': {'openIdConnectUrl': 'https://iam.ascend.icsgate.eu/realms/esa-maap/.well-known/openid-configuration', 'type': 'openIdConnect'}}, 'processing:datetime': '2025-10-02T02:38:47.000Z', 'eopf:instrument_mode': 'DEFAULT', 'updated': '2025-10-14T07:05:47Z', 'sat:absolute_orbit': 7642, 'product:acquisition_type': 'other'}
  - Assets: ['thumbnail', 'product', 'metadata_ogc_10_157r4', 'enclosure_h5', 'metadata_ogc_17_003r2', 'metadata_iso_19139', 'enclosure_hdr', 'quicklook']

Item ID: ECA_EXBA_MSI_COP_2A_20251007T134713Z_20251008T162223Z_07727E
  - Datetime: 2025-10-07 13:47:13+00:00
  - BBOX: [-3.97911, -22.715141, 6.304434, 22.59963]
  - Properties: {'start_datetime': '2025-10-07T13:47:13.000Z', 'end_datetime': '2025-10-07T13:58:49.000Z', 'processing:facility': 'PDGS-CPFxx', 'product:type': 'MSI_COP_2A', 'sat:anx_datetime': '2025-10-07T13:06:49.316Z', 'title': 'ECA_EXBA_MSI_COP_2A_20251007T134713Z_20251008T162223Z_07727E', 'platform': 'EarthCARE', 'datetime': '2025-10-07T13:47:13.000Z', 'instruments': ['MSI'], 'constellation': 'EarthCARE', 'sat:orbit_state': 'descending', 'processing:software': {'M-CLD': '11.30'}, 'eof-eos:mission_phase': 'Routine', 'grid:code': 'WRS-7727-E', 'processing:level': 'L2A', 'sci:doi': '10.57780/eca-94b4e53', 'created': '2025-10-08T16:22:23.000Z', 'published': '2025-10-12T07:28:10.143Z', 'version': 'BA', 'auth:schemes': {'s3': {'type': 's3'}, 'oidc': {'openIdConnectUrl': 'https://iam.ascend.icsgate.eu/realms/esa-maap/.well-known/openid-configuration', 'type': 'openIdConnect'}}, 'processing:datetime': '2025-10-08T16:22:23.000Z', 'eopf:instrument_mode': 'DEFAULT', 'updated': '2025-10-13T17:24:05Z', 'sat:absolute_orbit': 7727, 'product:acquisition_type': 'other'}
  - Assets: ['thumbnail', 'product', 'metadata_ogc_10_157r4', 'enclosure_h5', 'metadata_ogc_17_003r2', 'metadata_iso_19139', 'enclosure_hdr', 'quicklook']

# Choose one item
ec_product = items[1]

Results#

Assets in the ESA MAAP STAC Catalog: Each granule (one frame of EarthCARE data per product) includes multiple assets, which are different files that serve distinct purposes. These assets can include preview images, scientific data, metadata, and more.

# Optional: Inspect available assets
for asset_key, asset in ec_product.assets.items():
        print(f"  Asset key: {asset_key}")
        print(f"    href: {asset.href}")
        print(f"    type: {asset.media_type}")
        print(f"    roles: {asset.roles}")

  Asset key: thumbnail
    href: https://catalog.maap.eo.esa.int/data/earthcare-pdgs-01/EarthCARE/MSI_COP_2A/BA/2025/10/13/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/public/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.BID_1.jpeg
    type: image/jpeg
    roles: ['thumbnail']
  Asset key: product
    href: https://catalog.maap.eo.esa.int/data/zipper/earthcare-pdgs-01/EarthCARE/MSI_COP_2A/BA/2025/10/13/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E
    type: application/zip
    roles: ['data', 'metadata', 'archive']
  Asset key: metadata_ogc_10_157r4
    href: https://catalog.maap.eo.esa.int/catalogue/collections/EarthCAREL2Validated_MAAP/items/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E?httpAccept=application/gml%2Bxml&recordSchema=om
    type: application/gml+xml;profile="http://www.opengis.net/spec/EOMPOM/1.1"
    roles: ['metadata']
  Asset key: enclosure_h5
    href: https://catalog.maap.eo.esa.int/data/earthcare-pdgs-01/EarthCARE/MSI_COP_2A/BA/2025/10/13/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.h5
    type: application/x-hdf5
    roles: ['data']
  Asset key: metadata_ogc_17_003r2
    href: https://catalog.maap.eo.esa.int/catalogue/collections/EarthCAREL2Validated_MAAP/items/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E?mode=owc
    type: application/geo+json;profile="http://www.opengis.net/spec/eo-geojson/1.0"
    roles: ['metadata']
  Asset key: metadata_iso_19139
    href: https://catalog.maap.eo.esa.int/catalogue/collections/EarthCAREL2Validated_MAAP/items/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E?httpAccept=application/vnd.iso.19139%2Bxml
    type: application/vnd.iso.19139+xml
    roles: ['metadata']
  Asset key: enclosure_hdr
    href: https://catalog.maap.eo.esa.int/data/earthcare-pdgs-01/EarthCARE/MSI_COP_2A/BA/2025/10/13/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.HDR
    type: image/vnd.radiance
    roles: ['data']
  Asset key: quicklook
    href: https://catalog.maap.eo.esa.int/data/earthcare-pdgs-01/EarthCARE/MSI_COP_2A/BA/2025/10/13/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/public/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.BID_1.jpeg
    type: image/jpeg
    roles: ['overview']

Tips:

Want a quick look? Use the quicklook or thumbnail to preview the data.
Need to analyze? Work with the enclosure files as demonstrated in the scripts later on.
Don’t need everything? Avoid the .zip unless you really need to download all files.
Curious about metadata? Open the XML/JSON metadata files for detailed info.

Quicklook of the data#

You don’t need to authenticate or authorize to preview the data.
By referencing the thumbnail asset, you’re accessing a remote URL where a quicklook image of the product is stored. Note that not all products necessarily have a quicklook stored. This provides a fast and convenient way to visually inspect the data before downloading or processing it.

ql_url = ec_product.assets['thumbnail'].href
display(Image(url= ql_url))

Token#

You can generate an offline access token that is valid for 90 days here. We recommend to save this personal offline token in a separate file and not paste it into your scripts. This offline access token in combination with a client ID and secret creates a refresh token which authenticates/authorizes you to access the data files. In simple terms, the offline token is like a temporary key that helps you securely log in without needing to re-enter your credentials every time.

# offline token link: https://portal.maap.eo.esa.int/ini/services/auth/token/90dToken.php
CLIENT_ID="offline-token"
CLIENT_SECRET="p1eL7uonXs6MDxtGbgKdPVRAmnGxHpVE"
# Save your personal offline token in a text file token_yourname.txt in the same directory as this script

if pathlib.Path("token_yourname.txt").exists():
  with open("token_yourname.txt","rt") as f:
    OFFLINE_TOKEN = f.read().strip().replace("\n","")
    url = "https://iam.maap.eo.esa.int/realms/esa-maap/protocol/openid-connect/token"
    data = {
        "client_id": CLIENT_ID,
        "client_secret": CLIENT_SECRET,
        "grant_type": "refresh_token",
        "refresh_token": OFFLINE_TOKEN,
        "scope": "offline_access openid"
    }

    response = requests.post(url, data=data)
    response.raise_for_status()

    response_json = response.json()
    access_token = response_json.get('access_token')
    print("Access token retrieved successfully.")

    if not access_token:
        raise RuntimeError("Failed to retrieve access token from IAM response")

Access token retrieved successfully.

Stream and plot data#

⚠️ NEW: The .h5 file is now stored under enclosure_h5 while the header file is stored in the enclosure_hdr asset.

# Fetching the url of the desired file
ds_url = ec_product.assets.get('enclosure_h5').href  # Updated to enclosure_h5
print(f"Dataset URL: {ds_url}")

Dataset URL: https://catalog.maap.eo.esa.int/data/earthcare-pdgs-01/EarthCARE/MSI_COP_2A/BA/2025/10/13/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E/ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.h5

This provides a faster way to access the NetCDF data in the cloud. It configures how the HDF5 (.h5) files are read and cached.

io_params = {
    "fsspec_params": {
        "cache_type": "blockcache",
        "block_size": 8 * 1024 * 1024
    },
    "h5py_params": {
        "driver_kwds": {
            "rdcc_nbytes": 8 * 1024 * 1024
        }
    }
}

fs = fsspec.filesystem(
    "https", 
    headers={"Authorization": f"Bearer {access_token}"}, 
    **io_params["fsspec_params"]  )

# Open the file and read it into an xarray Dataset
with fs.open(ds_url, "rb") as f:
    ds = xr.open_dataset(f, 
                         engine="h5netcdf", 
                         **io_params["h5py_params"],  
                         group="ScienceData")
    
    # Do something with ds! Here we plot two variables as an example.
    fig, axes = plt.subplots(1, 2, figsize=(14, 6))

    # Plot Cloud Water Path
    ds["cloud_water_path"].plot(ax=axes[0], cmap="Blues")
    axes[0].set_title("Cloud Water Path")

    # Plot Cloud Top Temperature
    ds["cloud_top_temperature"].plot(ax=axes[1], cmap="plasma")
    axes[1].set_title("Cloud Top Temperature")

    plt.tight_layout()
    plt.show()
    

../_images/cb36f2eb2cc3699feb0d2ad52d60da75052c6b74b93a76753784ac628e0b6833.png

Download data#

You can also use your token and the url to download data and not just stream it.

def download_file_with_bearer_token(url, token, disable_bar=False):
  """
  Downloads a file from a given URL using a Bearer token.
  """

  try:
    headers = {"Authorization": f"Bearer {token}"}
    response = requests.get(url, headers=headers, stream=True)
    response.raise_for_status()  # Raise an exception for bad status codes
    file_size = int(response.headers.get('content-length', 0))

    chunk_size = 8 * 1024 * 1024 # Byes - 1MiB
    file_path = url.rsplit('/', 1)[-1] 
    print(file_path)
    with open(file_path, "wb") as f, tqdm(
        desc=file_path,
        total=file_size,
        unit='iB',
        unit_scale=True,
        unit_divisor=1024,
        disable=disable_bar,
      ) as bar:
      for chunk in response.iter_content(chunk_size=chunk_size):
        read_size=f.write(chunk)
        bar.update(read_size)

    if (disable_bar): 
      print(f"File downloaded successfully to {file_path}")

  except requests.exceptions.RequestException as e:
    print(f"Error downloading file: {e}")

import requests
download_file_with_bearer_token(ds_url, access_token)

ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.h5

ECA_EXBA_MSI_COP_2A_20251013T131444Z_20251013T162407Z_07820E.h5: 100%|██████████| 131M/131M [00:38<00:00, 3.57MiB/s] 