Merge pull request #1061 from betolink/fsspec

* fsspec data fetching improvements and exposing the kwargs so users can tweak it further
* retry logic if a download fails.
* multi-threaded download in the cloud (fixed)
* show_progress=True by default if the session is interactive

Author: betolink

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Changed
 
+- Change default cache behavior in fsspec from `readahead` to `blockcache`.
+  Allow user defined config with `open_kwargs` in the `.open()` method.
+  This improves performance by an order of magnitude.
+  ([#251](https://github.com/nsidc/earthaccess/discussions/251))([#771](https://github.com/nsidc/earthaccess/discussions/771))
+  ([@betolink](https://github.com/betolink))
 - Add `show_progress` argument to `earthaccess.download()` to let the user control display of progress bars. Defaults to true for interactive sessions, otherwise false.
   ([#612](https://github.com/nsidc/earthaccess/issues/612))
   ([#1065](https://github.com/nsidc/earthaccess/pull/1065))
@@ -25,6 +30,9 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Added
 
+- Added `tenacity` to retry downloads up to 3 times with exponential backoff time, replaces #1016
+  ([#481](https://github.com/nsidc/earthaccess/issues/481))
+  ([@betolink](https://github.com/betolink))
 - Add notebook demonstrating workflow with TEMPO Level 3 data as a virtual dataset
   ([#924](https://github.com/nsidc/earthaccess/pull/924))
   ([@danielfromearth](https://github.com/danielfromearth))
@@ -60,6 +68,7 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Fixed
 
+- Files can be downloaded in the cloud([#1009](https://github.com/nsidc/earthaccess/issues/1009))([betolink](https://github.com/betolink))
 - Corrected Harmony typo in notebooks/Demo.ipynb([#995](https://github.com/nsidc/earthaccess/issues/995))([stelios-c](https://github.com/stelios-c))
 - Resolved an error in virtual dataset tutorial notebook ([#1044](https://github.com/nsidc/earthaccess/issues/1044))([danielfromearth](https://github.com/danielfromearth))
 - Issue when `FileDistributionInformation` did not exist for a collection

docs/user_guide/fsspec.ipynb

@@ -284,6 +284,7 @@ def download(
     threads: int = 8,
     *,
     show_progress: Optional[bool] = None,
+    credentials_endpoint: Optional[str] = None,
     pqdm_kwargs: Optional[Mapping[str, Any]] = None,
 ) -> List[Path]:
     """Retrieves data granules from a remote storage system. Provide the optional `local_path` argument to prevent repeated downloads.
@@ -300,6 +301,8 @@ def download(
             month, and day of the current date, and `UUID` is the last 6 digits
             of a UUID4 value.
         provider: if we download a list of URLs, we need to specify the provider.
+        credentials_endpoint: S3 credentials endpoint to be used for obtaining temporary S3 credentials. This is only required if
+            the metadata doesn't include it, or we pass urls to the method instead of `DataGranule` instances.
         threads: parallel number of threads to use to download the files, adjust as necessary, default = 8
         show_progress: whether or not to display a progress bar. If not specified, defaults to `True` for interactive sessions
             (i.e., in a notebook or a python REPL session), otherwise `False`.
@@ -326,6 +329,7 @@ def download(
             local_path,
             provider,
             threads,
+            credentials_endpoint=credentials_endpoint,
             show_progress=show_progress,
             pqdm_kwargs=pqdm_kwargs,
         )
@@ -341,8 +345,10 @@ def open(
     granules: Union[List[str], List[DataGranule]],
     provider: Optional[str] = None,
     *,
+    credentials_endpoint: Optional[str] = None,
     show_progress: Optional[bool] = None,
     pqdm_kwargs: Optional[Mapping[str, Any]] = None,
+    open_kwargs: Optional[Dict[str, Any]] = None,
 ) -> List[AbstractFileSystem]:
     """Returns a list of file-like objects that can be used to access files
     hosted on S3 or HTTPS by third party libraries like xarray.
@@ -356,15 +362,19 @@ def open(
         pqdm_kwargs: Additional keyword arguments to pass to pqdm, a parallel processing library.
             See pqdm documentation for available options. Default is to use immediate exception behavior
             and the number of jobs specified by the `threads` parameter.
+        open_kwargs: Additional keyword arguments to pass to `fsspec.open`, such as `cache_type` and `block_size`.
+            Defaults to using `blockcache` with a block size determined by the file size (4 to 16MB).
 
     Returns:
         A list of "file pointers" to remote (i.e. s3 or https) files.
     """
     return earthaccess.__store__.open(
         granules=granules,
         provider=_normalize_location(provider),
+        credentials_endpoint=credentials_endpoint,
         show_progress=show_progress,
         pqdm_kwargs=pqdm_kwargs,
+        open_kwargs=open_kwargs,
     )