Issues running bm_raster on Anaconda

[guerreroda]

Bug Report

Description

The issue arises when replicating the daily data example.
There are no problems up to gdf.explore(). But the bm_raster() function fails to process the files.

Reproducibility

This is the code:

gdf = geopandas.read_file(
    "https://geodata.ucdavis.edu/gadm/gadm4.1/json/gadm41_BHS_1.json.zip"
)
gdf.explore()

bearer = os.getenv(BLACKMARBLE_TOKEN)

r_20210205 = bm_raster(
    gdf, product_id="VNP46A2", date_range="2021-02-05", bearer=BLACKMARBLE_TOKEN
)

The output:

GETTING MANIFEST...: 100%
2/2 [00:00<00:00, 2.77it/s]
QUEUEING TASKS | Downloading...: 100%
2/2 [00:00<00:00, 331.33it/s]
PROCESSING TASKS | Downloading...: 100%
2/2 [00:35<00:00, 15.54s/it]

[2024-05-17 16:14:14 - backoff:105 - INFO] Backing off _download_file(...) for 0.7s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:14 - backoff:105 - INFO] Backing off _download_file(...) for 0.4s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:19 - backoff:105 - INFO] Backing off _download_file(...) for 0.1s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:19 - backoff:105 - INFO] Backing off _download_file(...) for 0.1s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:25 - backoff:105 - INFO] Backing off _download_file(...) for 0.8s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:25 - backoff:105 - INFO] Backing off _download_file(...) for 1.5s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:31 - backoff:105 - INFO] Backing off _download_file(...) for 5.3s (httpx.ReadTimeout: The read operation timed out)
[2024-05-17 16:14:31 - backoff:105 - INFO] Backing off _download_file(...) for 7.0s (httpx.ReadTimeout: The read operation timed out)

COLLECTING RESULTS | Downloading...: 100%
2/2 [00:00<00:00, 399.69it/s]
COLLATING RESULTS | Processing...: 0%
0/1 [00:00<?, ?it/s]

---------------------------------------------------------------------------
CPLE_BaseError                            Traceback (most recent call last)
File rasterio\\crs.pyx:775, in rasterio.crs.CRS.from_user_input()

File rasterio\\_err.pyx:209, in rasterio._err.exc_wrap_ogrerr()

CPLE_BaseError: OGR Error code 6

During handling of the above exception, another exception occurred:

CRSError                                  Traceback (most recent call last)
Cell In[9], line 4
      1 BLACKMARBLE_TOKEN = "[SUPRESSED]"
      2 bearer = os.getenv(BLACKMARBLE_TOKEN)
----> 4 r_20210205 = bm_raster(
      5     gdf, product_id="VNP46A2", date_range="2021-02-05", bearer=BLACKMARBLE_TOKEN
      6 )

File ~\anaconda3\Lib\site-packages\pydantic\validate_call_decorator.py:59, in validate_call.<locals>.validate.<locals>.wrapper_function(*args, **kwargs)
     57 @functools.wraps(function)
     58 def wrapper_function(*args, **kwargs):
---> 59     return validate_call_wrapper(*args, **kwargs)

File ~\anaconda3\Lib\site-packages\pydantic\_internal\_validate_call.py:81, in ValidateCallWrapper.__call__(self, *args, **kwargs)
     80 def __call__(self, *args: Any, **kwargs: Any) -> Any:
---> 81     res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
     82     if self.__return_pydantic_validator__:
     83         return self.__return_pydantic_validator__(res)

File ~\anaconda3\Lib\site-packages\blackmarble\raster.py:283, in bm_raster(gdf, product_id, date_range, bearer, variable, quality_flag_rm, check_all_tiles_exist, file_directory, file_prefix, file_skip_if_exists)
    279 filenames = _pivot_paths_by_date(pathnames).get(date)
    281 try:
    282     # Open each GeoTIFF file as a DataArray and store in a list
--> 283     da = [
    284         rioxarray.open_rasterio(
    285             h5_to_geotiff(
    286                 f,
    287                 variable=variable,
    288                 quality_flag_rm=quality_flag_rm,
    289                 output_prefix=file_prefix,
    290                 output_directory=d,
    291             ),
    292         )
    293         for f in filenames
    294     ]
    295     ds = merge_arrays(da)
    296     ds = ds.rio.clip(gdf.geometry.apply(mapping), gdf.crs, drop=True)

File ~\anaconda3\Lib\site-packages\blackmarble\raster.py:285, in <listcomp>(.0)
    279 filenames = _pivot_paths_by_date(pathnames).get(date)
    281 try:
    282     # Open each GeoTIFF file as a DataArray and store in a list
    283     da = [
    284         rioxarray.open_rasterio(
--> 285             h5_to_geotiff(
    286                 f,
    287                 variable=variable,
    288                 quality_flag_rm=quality_flag_rm,
    289                 output_prefix=file_prefix,
    290                 output_directory=d,
    291             ),
    292         )
    293         for f in filenames
    294     ]
    295     ds = merge_arrays(da)
    296     ds = ds.rio.clip(gdf.geometry.apply(mapping), gdf.crs, drop=True)

File ~\anaconda3\Lib\site-packages\blackmarble\raster.py:129, in h5_to_geotiff(f, variable, quality_flag_rm, output_directory, output_prefix)
    121 height, width = data.shape
    122 transform = from_origin(
    123     left,
    124     top,
    125     (right - left) / width,
    126     (top - bottom) / height,
    127 )
--> 129 with rasterio.open(
    130     output_path,
    131     "w",
    132     driver="GTiff",
    133     height=height,
    134     width=width,
    135     count=1,
    136     dtype=data.dtype,
    137     crs="EPSG:4326",
    138     transform=transform,
    139 ) as dst:
    140     dst.write(data, 1)
    141     dst.update_tags(**attrs)

File ~\anaconda3\Lib\site-packages\rasterio\env.py:451, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    448     session = DummySession()
    450 with env_ctor(session=session):
--> 451     return f(*args, **kwds)

File ~\anaconda3\Lib\site-packages\rasterio\__init__.py:327, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    325 writer = get_writer_for_driver(driver)
    326 if writer is not None:
--> 327     dataset = writer(
    328         path,
    329         mode,
    330         driver=driver,
    331         width=width,
    332         height=height,
    333         count=count,
    334         crs=crs,
    335         transform=transform,
    336         dtype=dtype,
    337         nodata=nodata,
    338         sharing=sharing,
    339         **kwargs
    340     )
    341 else:
    342     raise DriverCapabilityError(
    343         "Writer does not exist for driver: %s" % str(driver)
    344     )

File rasterio\\_io.pyx:1563, in rasterio._io.DatasetWriterBase.__init__()

File rasterio\\_io.pyx:1592, in rasterio._io.DatasetWriterBase._set_crs()

File rasterio\\crs.pyx:777, in rasterio.crs.CRS.from_user_input()

CRSError: The WKT could not be parsed. OGR Error code 6

Environment

• Operating System: Windows 11
• Additional Environment Details: Jupyter Lab, Anaconda3,

I would appreciate any advice.

[g4brielvs]

@guerreroda Thank you for using BlackMarblePy and reporting the issue. Could you please check the gdf.crs?

[guerreroda]

@g4brielvs Thanks for your help. It's been a while so I had to request the token again. The error has changed but I am running the same code (from the example):

my_token = "SECURE"


# https://worldbank.github.io/blackmarblepy/notebooks/blackmarblepy.html

import colorcet as cc
import contextily as cx
import geopandas
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

from blackmarble.extract import bm_extract
from blackmarble.raster import bm_raster

%load_ext autoreload
%autoreload 2

plt.rcParams["figure.figsize"] = (18, 10)

gdf = geopandas.read_file(
    "https://geodata.ucdavis.edu/gadm/gadm4.1/json/gadm41_VEN_1.json.zip"
)
gdf.explore()

r_20210205 = bm_raster(
    gdf, product_id="VNP46A2", date_range="2023-02-05", bearer=my_token
)

I have also tried bearer=bearer but the error says bearer is a string.

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\guerr\\AppData\\Local\\Temp\\tmpomuonw9j\\VNP46A2.A2023036.h10v07.001.2023044152210.tif'

I also get several warnings of these:

Error displaying widget

The gdf.crs:

<Geographic 2D CRS: EPSG:4326>
Name: WGS 84
Axis Info [ellipsoidal]:
- Lat[north]: Geodetic latitude (degree)
- Lon[east]: Geodetic longitude (degree)
Area of Use:
- name: World.
- bounds: (-180.0, -90.0, 180.0, 90.0)
Datum: World Geodetic System 1984 ensemble
- Ellipsoid: WGS 84
- Prime Meridian: Greenwich

[g4brielvs]

@guerreroda Thank you for reporting this issue. It seems to a Windows-specific behavior, which will be addressed in the next release. In the meantime, is this intermittent or is it occurring on every execution?