Issue with downloading Satellite Albedo Data for smaller areas and time Periods

[FaezehKarimian]

Notebook description

Hi @malmans2 ,
I am currently working with the [satellite-albedo] dataset, and I’ve encountered an issue when attempting to download data for smaller areas and time periods. Specifically, I’m able to successfully retrieve data for the region defined by the coordinates 46.91, 50.75, 46.81, 50.85, but when I try to change the area, I receive the following error message: "HTTPError: 403 Client Error: Forbidden for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/satellite-albedo/execution
Cost limits exceeded
Your request is too large, please reduce your selection."
I’ve checked the parameters, and I believe the issue may be related to the selection size or data volume. Could you provide some guidance on how to resolve this issue?

Thank you for your assistance. I look forward to your response.

Notebook link or upload

Albedo_timeseries_code.zip

Anything else we need to know?

No response

Environment

Details

[malmans2]

Hi there,

I contacted ECMWF about this. It looks like this is not the expected behaviour, so they might need to change something in the dataset forms. I will let you know as soon as I have new info.

[malmans2]

@FaezehKarimian good news, they fixed it in the CDS.
As you can see from the forms, the area widget is now available in the download form: https://cds.climate.copernicus.eu/datasets/satellite-albedo?tab=download

Can you try again? Your code should work again now.

[FaezehKarimian]

@FaezehKarimian good news, they fixed it in the CDS. As you can see from the forms, the area widget is now available in the download form: https://cds.climate.copernicus.eu/datasets/satellite-albedo?tab=download

Can you try again? Your code should work again now.

Thank Mattia, but the problem still persists. I receive this warning : "satellite='spot'
2025-01-29 10:11:46,058 INFO [2024-09-26T00:00:00] Watch our Forum for Announcements, news and other discussed topics.
2025-01-29 10:11:46,059 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
2025-01-29 10:11:46,340 INFO [2024-09-26T00:00:00] Watch our Forum for Announcements, news and other discussed topics.
2025-01-29 10:11:46,340 WARNING [2024-06-16T00:00:00] CDS API syntax is changed and some keys or parameter names may have also changed. To avoid requests failing, please use the "Show API request code" tool on the dataset Download Form to check you are using the correct syntax for your API request.
2025-01-29 10:11:46,487 INFO Request ID is 991c3800-ce3e-41ca-a8c9-96a5e84a9127
2025-01-29 10:11:46,539 INFO status has been updated to accepted

Albedo_timeseries_code (1).zip

"

[malmans2]

That's not a problem. The CDS is warning users that the API syntax might have changed.
But you request is not impacted by those changes, and it has been successfully accepted.
The job status will become running and (hopefully) succesfull.

Anyway, if you want me to download the data for you I can do that.

[FaezehKarimian]

Yes Mattia, thanks, that would be a great help because I think it takes a lot of time on my end. Could you estimate how long it would take to download albedo data for 10 years? I need the albedo variable 'albb_bh' for these two areas: [13.2, 77.4, 12.8, 77.8] and [44.6, 11.2, 44.2, 11.4], for the following years: SPOT (1999–2013) and PROBA (2014–2020).

[malmans2]

I don't know. I don't have any priority on the CDS queue, and I'm downloading data for other evaluators as well.
Usually the queue is shorter overnight, so hopefully they'll be ready tomorrow morning.

[FaezehKarimian]

I don't know. I don't have any priority on the CDS queue, and I'm downloading data for other evaluators as well. Usually the queue is shorter overnight, so hopefully they'll be ready tomorrow morning.

That sounds great, thank you very much.

[malmans2]

Hi @FaezehKarimian,

The data you need is cached on the VM.
Make sure you start from the snippet below. I used "region1" and "region2", but you can choose better names for the regions. I have not applied any transform function as I don't know exactly what you need. Let me know if you need further help.

Here is the code to retrieve the data:

import collections

import xarray as xr
from c3s_eqc_automatic_quality_control import download

collection_id = "satellite-albedo"
chunks = {"year": 1}
common_request = {
    "variable": "albb_bh",
    "product_version": "v2",
    "month": [f"{month:02d}" for month in range(1, 13)],
    "nominal_day": "10",
}
requests = {
    "spot": {
        "year": [str(year) for year in range(1999, 2014)],
        "horizontal_resolution": "1km",
        "sensor": "vgt",
    },
    "proba": {
        "year": [str(year) for year in range(2014, 2021)],
        "horizontal_resolution": "1km",
        "sensor": "vgt",
    },
}
areas = {
    "region1": [13.2, 77.4, 12.8, 77.8],
    "region2": [44.6, 11.2, 44.2, 11.4],
}

datasets = collections.defaultdict(list)
for region, area in areas.items():
    for satellite, request in requests.items():
        print(f"{satellite=} {region=}")
        ds = download.download_and_transform(
            collection_id,
            common_request | request | {"satellite": satellite, "area": area},
            chunks=chunks,
            drop_variables=["crs"],
        )
        datasets[region].append(ds.expand_dims(region=[region], satellite=[satellite]))
datasets = {k: xr.concat(v, "satellite") for k, v in datasets.items()}

[FaezehKarimian]

Hi @FaezehKarimian,

The data you need is cached on the VM. Make sure you start from the snippet below. I used "region1" and "region2", but you can choose better names for the regions. I have not applied any transform function as I don't know exactly what you need. Let me know if you need further help.

Here is the code to retrieve the data:

import collections

import xarray as xr
from c3s_eqc_automatic_quality_control import download

collection_id = "satellite-albedo"
chunks = {"year": 1}
common_request = {
"variable": "albb_bh",
"product_version": "v2",
"month": [f"{month:02d}" for month in range(1, 13)],
"nominal_day": "10",
}
requests = {
"spot": {
"year": [str(year) for year in range(1999, 2014)],
"horizontal_resolution": "1km",
"sensor": "vgt",
},
"proba": {
"year": [str(year) for year in range(2014, 2021)],
"horizontal_resolution": "1km",
"sensor": "vgt",
},
}
areas = {
"region1": [13.2, 77.4, 12.8, 77.8],
"region2": [44.6, 11.2, 44.2, 11.4],
}

datasets = collections.defaultdict(list)
for region, area in areas.items():
for satellite, request in requests.items():
print(f"{satellite=} {region=}")
ds = download.download_and_transform(
collection_id,
common_request | request | {"satellite": satellite, "area": area},
chunks=chunks,
drop_variables=["crs"],
)
datasets[region].append(ds.expand_dims(region=[region], satellite=[satellite]))
datasets = {k: xr.concat(v, "satellite") for k, v in datasets.items()}

Hi @malmans2 , I was able to successfully download the albedo data. I truly appreciate your help, thank you so much!

[malmans2]

Hi @FaezehKarimian,

I received an email notification but it looks like the message disappeared on GitHub. Did you delete the message? Do you still need support?

[FaezehKarimian]

Hi @FaezehKarimian,

I received an email notification but it looks like the message disappeared on GitHub. Did you delete the message? Do you still need support?

Hi, yes @malmans2 , I wanted to ask if you can download albedo data for me? I wanted to change the coordinate, so I deleted the message, but now the coordinates are:

#Salerno, Urban: [40.42, 14.44, 40.39, 14.47]
#Salerno, Rural: [40.65, 14.50, 40.35, 14.80]
#Spain, Urban: [37.30, 03.20, 37.15, 03.40]
#Spain, Rural: [37.45, 03.30, 37.40, 03.40]
#U.S., Urban: [33.50, 112.30, 33.10, 113.50]
#U.S., Rural: [34.20, 111.50, 33.80, 112.00]

[malmans2]

OK, I'll let you know when they're ready