diff --git a/HISTORY.md b/HISTORY.md index 5a05d913..1b21afee 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,6 +1,6 @@ # cloudpathlib Changelog -## UNRELEASED +## v0.25.0 (2026-08-22) - **Security fix ([GHSA-r4f8-3xc4-c8vw](https://github.com/drivendataorg/cloudpathlib/security/advisories/GHSA-r4f8-3xc4-c8vw)): local path traversal via `..` in cloud object keys.** Cloud object keys are opaque strings and some backends (e.g. Google Cloud Storage) accept `..` @@ -14,16 +14,18 @@ and `CloudPath.copytree` (including joins that can escape via `\` or drive letters in keys on Windows), and the `cloudpathlib.local` mock's storage-side mapping. Reported by mohammad adnan (cystack.ps redteam). +- Added a `lazy` keyword argument to `CloudPath.walk`. By default (`lazy=False`) the existing fast behavior is preserved: the whole subtree is fetched up front with a single recursive listing. Passing `lazy=True` lists each directory on demand so that, when `top_down=True`, callers can prune subdirectories by modifying `dirnames` in-place (à la `os.walk` / `Path.walk`) to skip fetching the contents of those subtrees entirely — dramatically reducing API calls for large, sparsely-traversed trees. (Issue [#518](https://github.com/drivendataorg/cloudpathlib/issues/518), PR [#574](https://github.com/drivendataorg/cloudpathlib/pull/574)) +- Added presigned URL support to the mock clients in `cloudpathlib.local`, so `as_url(presign=True)` on a `LocalS3Path`, `LocalGSPath`, or `LocalAzureBlobPath` returns a deterministic URL with `expires` and `signature` query parameters instead of raising `NotImplementedError`. This lets code that generates presigned URLs be exercised in tests that use the mock classes. (PR [#572](https://github.com/drivendataorg/cloudpathlib/pull/572)) +- Fixed S3 copy and move operations to forward copy-specific extra args such as `CopySourceSSECustomerKey`, and added `addressing_style="virtual"` support for `S3Client` (Issues [#500](https://github.com/drivendataorg/cloudpathlib/issues/500), [#527](https://github.com/drivendataorg/cloudpathlib/issues/527), PR [#576](https://github.com/drivendataorg/cloudpathlib/pull/576)) +- Changed `S3Client._get_metadata` to read object metadata with `HeadObject` instead of `GetObject`, so `stat`, `etag`, and `size` no longer open the object body. Also fixes a `KeyError` on `ContentLength` against S3-compatible gateways that drop `Content-Length` from `GetObject` responses. (Issue [#564](https://github.com/drivendataorg/cloudpathlib/issues/564), PR [#565](https://github.com/drivendataorg/cloudpathlib/pull/565)) +- Fixed `KeyError` on `ETag` against S3-compatible gateways that omit `ETag` from `HeadObject` responses. (Issue [#582](https://github.com/drivendataorg/cloudpathlib/issues/582), PR [#583](https://github.com/drivendataorg/cloudpathlib/pull/583)) +- Fixed mypy 2.x type errors in `Client` and `CloudPath` that caused CI lint failures (Issue [#563](https://github.com/drivendataorg/cloudpathlib/issues/563), PR [#566](https://github.com/drivendataorg/cloudpathlib/pull/566)) - Added `AGENTS.md` with repository-specific guidance for coding agents covering contributor workflow, compatibility expectations, test rig and mock usage, live backend validation, and PR - hygiene. -- Fixed mypy 2.x type errors in `Client` and `CloudPath` that caused CI lint failures (Issue [#563](https://github.com/drivendataorg/cloudpathlib/issues/563), PR [#566](https://github.com/drivendataorg/cloudpathlib/pull/566)) -- Changed `S3Client._get_metadata` to read object metadata with `HeadObject` instead of `GetObject`, so `stat`, `etag`, and `size` no longer open the object body. Also fixes a `KeyError` on `ContentLength` against S3-compatible gateways that drop `Content-Length` from `GetObject` responses. (Issue [#564](https://github.com/drivendataorg/cloudpathlib/issues/564), PR [#565](https://github.com/drivendataorg/cloudpathlib/pull/565)) -- Added a `lazy` keyword argument to `CloudPath.walk`. By default (`lazy=False`) the existing fast behavior is preserved: the whole subtree is fetched up front with a single recursive listing. Passing `lazy=True` lists each directory on demand so that, when `top_down=True`, callers can prune subdirectories by modifying `dirnames` in-place (à la `os.walk` / `Path.walk`) to skip fetching the contents of those subtrees entirely — dramatically reducing API calls for large, sparsely-traversed trees. (Issue [#518](https://github.com/drivendataorg/cloudpathlib/issues/518)) -- Fixed S3 copy and move operations to forward copy-specific extra args such as `CopySourceSSECustomerKey`, and added `addressing_style="virtual"` support for `S3Client` (Issues [#500](https://github.com/drivendataorg/cloudpathlib/issues/500), [#527](https://github.com/drivendataorg/cloudpathlib/issues/527)) -- Fix `KeyError` on `ETag` against S3-compatible gateways that omit `ETag` from `HeadObject` responses. (Issue [#582](https://github.com/drivendataorg/cloudpathlib/issues/582), PR [#583](https://github.com/drivendataorg/cloudpathlib/pull/583)) + hygiene. (PR [#573](https://github.com/drivendataorg/cloudpathlib/pull/573)) ## v0.24.0 (2026-04-29) + - Added support for S3 Multi-Region Access Point (MRAP) URLs in `S3Path` (Issue [#556](https://github.com/drivendataorg/cloudpathlib/issues/556), PR [#557](https://github.com/drivendataorg/cloudpathlib/pull/557)) - Added support for Pydantic serialization (Issue [#537](https://github.com/drivendataorg/cloudpathlib/issues/537), PR [#538](https://github.com/drivendataorg/cloudpathlib/pull/538)) diff --git a/cloudpathlib/s3/s3client.py b/cloudpathlib/s3/s3client.py index db34177d..e6aba796 100644 --- a/cloudpathlib/s3/s3client.py +++ b/cloudpathlib/s3/s3client.py @@ -72,7 +72,8 @@ def __init__( endpoint_url (Optional[str]): S3 server endpoint URL to use for the constructed boto3 S3 resource and client. Parameterize it to access a customly deployed S3-compatible object store such as MinIO, Ceph or any other. addressing_style (Optional[str]): S3 addressing style to pass through to boto3's S3 config. - Use `"virtual"` for virtual-hosted style URLs. + Use `"virtual"` for virtual-hosted style URLs; boto3's other values (`"path"` and + `"auto"`) are accepted as well. If None, boto3's default behavior is used. boto3_transfer_config (Optional[dict]): Instantiated TransferConfig for managing [s3 transfers](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig) content_type_method (Optional[Callable]): Function to call to guess media type (mimetype) when diff --git a/docs/docs/authentication.md b/docs/docs/authentication.md index bb10906f..bbf11a99 100644 --- a/docs/docs/authentication.md +++ b/docs/docs/authentication.md @@ -169,7 +169,19 @@ The args supported for downloads are the same as `boto3.s3.transfer.S3Transfer.A To use any of these extra args, pass them as a dict to `extra_args` when instantiating and `S3Client`. -Copy operations use `boto3.s3.transfer.S3Transfer.ALLOWED_COPY_ARGS`, including the `CopySourceSSECustomer*` args used for SSE-C. +Copy operations (used by `copy`, `copy_into`, `move`, and `move_into` when the source and destination share the same client, so the data is copied server-side rather than downloaded and re-uploaded) use `boto3.s3.transfer.S3Transfer.ALLOWED_COPY_ARGS`. That set includes most of the upload args above plus a handful of copy-specific args, which as of the time of writing are: + + - `CopySourceIfMatch` + - `CopySourceIfModifiedSince` + - `CopySourceIfNoneMatch` + - `CopySourceIfUnmodifiedSince` + - `CopySourceSSECustomerAlgorithm` + - `CopySourceSSECustomerKey` + - `CopySourceSSECustomerKeyMD5` + - `MetadataDirective` + - `TaggingDirective` + +If you use SSE-C, note that the `CopySourceSSECustomer*` args describe the _source_ object's key, while the plain `SSECustomer*` args describe the _destination_, so you generally need both sets in `extra_args` for copies and moves to work. ```python from cloudpathlib import S3Client diff --git a/docs/docs/caching.ipynb b/docs/docs/caching.ipynb index f2be92c5..48ae6aef 100644 --- a/docs/docs/caching.ipynb +++ b/docs/docs/caching.ipynb @@ -391,7 +391,7 @@ "source": [ "## Handling conflicts\n", "\n", - "We try to be conservative in terms of not losing data—especially data stored on the cloud, which is likely to be the canonical version. Given this, we will raise exceptions in two scenarios:\n", + "We try to be conservative in terms of not losing data—especially data stored on the cloud, which is likely to be the canonical version. Given this, we will raise exceptions in the following scenarios:\n", "\n", "`OverwriteNewerLocalError`\n", "This exception is raised if we are asked to download a file, but our local version in the cache is newer. This likely means that the cached version has been updated, but not pushed to the cloud. To work around this you could remove the cache version explicitly if you _know_ you don't need that data. If you did write changes you need, make sure your code uses the `cloudpathlib` versions of the `open`, `write_text`, or `write_bytes` methods, which will upload your changes to the cloud automatically.\n", @@ -406,7 +406,39 @@ "The `CloudPath.open` method supports a `force_overwrite_to_cloud` kwarg to force overwriting the cloud version.\n", "\n", "You can make overwriting the cloud copy with the local one being uploaded by setting the environment variable `CLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD=1` or `CLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD=True`.\n", - "\n" + "\n", + "`CloudPathLocalPathTraversalError`\n", + "Cloud object keys are opaque strings, and some backends will happily store a key that contains `..` segments. Since `cloudpathlib` mirrors keys onto local paths, such a key could otherwise resolve to a location outside of the cache directory (or outside of the destination directory you asked us to download into). We check every local path we compute and raise this exception instead of reading or writing outside the intended directory. This applies to anything that uses the cache (for example `open`, `read_text`, `read_bytes`, `write_text`, `write_bytes`, and `fspath`) as well as to `download_to`, `copy`, `move`, and `copytree` when the destination is a local path. If you hit this, the object key itself is unusual; use `CloudPath.read_bytes` or an explicit destination filename you control rather than mirroring the key onto disk.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Avoiding unnecessary listings when walking\n", + "\n", + "Caching saves you from re-downloading file _contents_, but simply listing a large tree can also cost a lot of API calls. `CloudPath.walk` gives you control over that tradeoff.\n", + "\n", + "By default (`lazy=False`), the whole subtree is fetched up front with a single recursive listing. That is the fastest option when you are going to visit the entire tree anyway.\n", + "\n", + "Passing `lazy=True` defers listing until each directory is visited (one non-recursive listing per directory). That is slower for a full walk, but—just like `os.walk` and `pathlib.Path.walk`—it lets you prune `dirnames` in place when `top_down=True`, so the contents of skipped subtrees are never fetched at all:\n", + "\n", + "```python\n", + "from cloudpathlib import CloudPath\n", + "\n", + "root = CloudPath(\"s3://ladi/Images\")\n", + "\n", + "for dirpath, dirnames, filenames in root.walk(lazy=True):\n", + " # we never list the contents of these subdirectories\n", + " dirnames[:] = [d for d in dirnames if not d.startswith(\"FEMA_CAP\")]\n", + "\n", + " for filename in filenames:\n", + " print(dirpath / filename)\n", + "```\n", + "\n", + "For large trees where you only need a few branches, pruning this way can reduce the number of API calls dramatically. If you are traversing everything, stick with the default.\n", + "\n", + "Note: `walk` also accepts `follow_symlinks` for signature parity with `pathlib`, but it has no effect—cloud object stores don't have symlinks.\n" ] }, { diff --git a/docs/docs/script/caching.py b/docs/docs/script/caching.py index 688f11f0..f42cf09a 100644 --- a/docs/docs/script/caching.py +++ b/docs/docs/script/caching.py @@ -129,18 +129,50 @@ # ## Handling conflicts # -# We try to be conservative in terms of not losing data—especially data stored on the cloud, which is likely to be the canonical version. Given this, we will raise exceptions in two scenarios: +# We try to be conservative in terms of not losing data—especially data stored on the cloud, which is likely to be the canonical version. Given this, we will raise exceptions in the following scenarios: # # `OverwriteNewerLocalError` # This exception is raised if we are asked to download a file, but our local version in the cache is newer. This likely means that the cached version has been updated, but not pushed to the cloud. To work around this you could remove the cache version explicitly if you _know_ you don't need that data. If you did write changes you need, make sure your code uses the `cloudpathlib` versions of the `open`, `write_text`, or `write_bytes` methods, which will upload your changes to the cloud automatically. # # The `CloudPath.open` method supports a `force_overwrite_from_cloud` kwarg to force overwriting your local version. # +# You can make overwriting the cache with the cloud copy the default by setting the environment variable `CLOUDPATHLIB_FORCE_OVERWRITE_FROM_CLOUD=1` or `CLOUDPATHLIB_FORCE_OVERWRITE_FROM_CLOUD=True`. +# # `OverwriteNewerCloudError` # This exception is raised if we are asked to upload a file, but the one on the cloud is newer than our local version. This likely means that a separate process has updated the cloud version, and we don't want to overwrite and lose that new data in the cloud. # # The `CloudPath.open` method supports a `force_overwrite_to_cloud` kwarg to force overwriting the cloud version. # +# You can make overwriting the cloud copy with the local one being uploaded by setting the environment variable `CLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD=1` or `CLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD=True`. +# +# `CloudPathLocalPathTraversalError` +# Cloud object keys are opaque strings, and some backends will happily store a key that contains `..` segments. Since `cloudpathlib` mirrors keys onto local paths, such a key could otherwise resolve to a location outside of the cache directory (or outside of the destination directory you asked us to download into). We check every local path we compute and raise this exception instead of reading or writing outside the intended directory. This applies to anything that uses the cache (for example `open`, `read_text`, `read_bytes`, `write_text`, `write_bytes`, and `fspath`) as well as to `download_to`, `copy`, `move`, and `copytree` when the destination is a local path. If you hit this, the object key itself is unusual; use `CloudPath.read_bytes` or an explicit destination filename you control rather than mirroring the key onto disk. +# + +# ## Avoiding unnecessary listings when walking +# +# Caching saves you from re-downloading file _contents_, but simply listing a large tree can also cost a lot of API calls. `CloudPath.walk` gives you control over that tradeoff. +# +# By default (`lazy=False`), the whole subtree is fetched up front with a single recursive listing. That is the fastest option when you are going to visit the entire tree anyway. +# +# Passing `lazy=True` defers listing until each directory is visited (one non-recursive listing per directory). That is slower for a full walk, but—just like `os.walk` and `pathlib.Path.walk`—it lets you prune `dirnames` in place when `top_down=True`, so the contents of skipped subtrees are never fetched at all: +# +# ```python +# from cloudpathlib import CloudPath +# +# root = CloudPath("s3://ladi/Images") +# +# for dirpath, dirnames, filenames in root.walk(lazy=True): +# # we never list the contents of these subdirectories +# dirnames[:] = [d for d in dirnames if not d.startswith("FEMA_CAP")] +# +# for filename in filenames: +# print(dirpath / filename) +# ``` +# +# For large trees where you only need a few branches, pruning this way can reduce the number of API calls dramatically. If you are traversing everything, stick with the default. +# +# Note: `walk` also accepts `follow_symlinks` for signature parity with `pathlib`, but it has no effect—cloud object stores don't have symlinks. # # ## Clearing the file cache diff --git a/docs/docs/script/patching_builtins.py b/docs/docs/script/patching_builtins.py index 51102c45..6e45ca89 100644 --- a/docs/docs/script/patching_builtins.py +++ b/docs/docs/script/patching_builtins.py @@ -3,6 +3,11 @@ # # Compatibility +#
+#

Experimental

+#

Patching open, os, os.path, and glob to work with CloudPath objects is experimental. It is off by default, and it may change or be removed in the future.

+#
+ # ## Patching Python builtins (third-party library compatibility) # # Not every Python library in the broad universe of Python libraries is implemented to accept pathlib-compatible objects like those implemented by cloudpathlib. Many libraries will only accept strings as filepaths. These libraries internally use `open`, functions from `os` and `os.path`, or other core library modules like `glob` to navigate paths and manipulate them. @@ -188,7 +193,6 @@ def library_function(filepath: str): from cloudpathlib import CloudPath, patch_all_builtins from PIL import Image - base = CloudPath("s3://cloudpathlib-test-bucket/patching_builtins/third_party/") img_path = base / "pillow_demo.png" diff --git a/docs/docs/script/testing_mocked_cloudpathlib.py b/docs/docs/script/testing_mocked_cloudpathlib.py index a389268d..4b9f07c3 100644 --- a/docs/docs/script/testing_mocked_cloudpathlib.py +++ b/docs/docs/script/testing_mocked_cloudpathlib.py @@ -19,6 +19,8 @@ # | Google Cloud Storage | `GSClient`
`GSPath` | `LocalGSClient`
`LocalGSPath` | `local_gs_implementation` | # | Amazon S3 | `S3Client`
`S3Path` | `LocalS3Client`
`LocalS3Path` | `local_s3_implementation` | # +# The `Local*` path classes also support presigned URLs. Calling `as_url(presign=True)` returns a deterministic stand-in URL with `expires` and `signature` query parameters (for example, `s3://mybucket/myfile.txt?expires=3600&signature=local`) instead of raising an error, so code that hands out presigned URLs can be exercised in your tests. The URL is not actually signed and can't be fetched; assert on its parts rather than trying to use it. +# # ## Examples: Monkeypatching in pytest @@ -78,5 +80,5 @@ # # Finally, the `reset_default_storage_dir` class method will clean up the current local storage temporary directory and set up a new one. We recommend you do this in the teardown of the test fixture. -get_ipython().run_cell_magic('run_pytest[clean]', '', '\nimport pytest\n\nfrom cloudpathlib import CloudPath, implementation_registry\nfrom cloudpathlib.local import LocalS3Client, LocalS3Path, local_s3_implementation\n\n\n@pytest.fixture\ndef cloud_asset_file(monkeypatch):\n """Fixture that patches CloudPath dispatch and also sets up test assets in LocalS3Client\'s\n local storage directory."""\n\n monkeypatch.setitem(implementation_registry, "s3", local_s3_implementation)\n\n # Option 1: Use LocalS3Path to set up test assets directly\n local_cloud_path = LocalS3Path("s3://cloudpathlib-test-bucket/altostratus.txt")\n local_cloud_path.write_text("altostratus")\n \n # Option 2: Use the pathlib.Path object that points to the local storage directory\n local_pathlib_path: Path = (\n LocalS3Client.get_default_storage_dir() / "cloudpathlib-test-bucket" / "nimbostratus.txt"\n )\n local_pathlib_path.parent.mkdir(exist_ok=True, parents=True)\n local_pathlib_path.write_text("nimbostratus")\n\n yield\n\n LocalS3Client.reset_default_storage_dir() # clean up temp directory and replace with new one\n\n\ndef test_with_assets(cloud_asset_file):\n """Testing that a patched CloudPath finds the test asset created in the fixture."""\n\n cloud_path_1 = CloudPath("s3://cloudpathlib-test-bucket/altostratus.txt")\n assert isinstance(cloud_path_1, LocalS3Path)\n assert cloud_path_1.exists()\n assert cloud_path_1.read_text() == "altostratus"\n \n cloud_path_2 = CloudPath("s3://cloudpathlib-test-bucket/nimbostratus.txt")\n assert isinstance(cloud_path_2, LocalS3Path)\n assert cloud_path_2.exists()\n assert cloud_path_2.read_text() == "nimbostratus"\n') +get_ipython().run_cell_magic('run_pytest[clean]', '', '\nimport pytest\n\nfrom cloudpathlib import CloudPath, implementation_registry\nfrom cloudpathlib.local import LocalS3Client, LocalS3Path, local_s3_implementation\n\n\n@pytest.fixture\ndef cloud_asset_file(monkeypatch):\n """Fixture that patches CloudPath dispatch and also sets up test assets in LocalS3Client\'s\n local storage directory."""\n\n monkeypatch.setitem(implementation_registry, "s3", local_s3_implementation)\n\n # Option 1: Use LocalS3Path to set up test assets directly\n local_cloud_path = LocalS3Path("s3://cloudpathlib-test-bucket/altostratus.txt")\n local_cloud_path.write_text("altostratus")\n\n # Option 2: Use the pathlib.Path object that points to the local storage directory\n local_pathlib_path: Path = (\n LocalS3Client.get_default_storage_dir() / "cloudpathlib-test-bucket" / "nimbostratus.txt"\n )\n local_pathlib_path.parent.mkdir(exist_ok=True, parents=True)\n local_pathlib_path.write_text("nimbostratus")\n\n yield\n\n LocalS3Client.reset_default_storage_dir() # clean up temp directory and replace with new one\n\n\ndef test_with_assets(cloud_asset_file):\n """Testing that a patched CloudPath finds the test asset created in the fixture."""\n\n cloud_path_1 = CloudPath("s3://cloudpathlib-test-bucket/altostratus.txt")\n assert isinstance(cloud_path_1, LocalS3Path)\n assert cloud_path_1.exists()\n assert cloud_path_1.read_text() == "altostratus"\n\n cloud_path_2 = CloudPath("s3://cloudpathlib-test-bucket/nimbostratus.txt")\n assert isinstance(cloud_path_2, LocalS3Path)\n assert cloud_path_2.exists()\n assert cloud_path_2.read_text() == "nimbostratus"\n') diff --git a/docs/docs/testing_mocked_cloudpathlib.ipynb b/docs/docs/testing_mocked_cloudpathlib.ipynb index e2eca0f8..7774acc2 100644 --- a/docs/docs/testing_mocked_cloudpathlib.ipynb +++ b/docs/docs/testing_mocked_cloudpathlib.ipynb @@ -30,7 +30,9 @@ "|:-|:-|:-|:-|\n", "| Azure Blob Storage | `AzureBlobClient`
`AzureBlobPath` | `LocalAzureBlobCient`
`LocalAzureBlobPath` | `local_azure_blob_implementation` |\n", "| Google Cloud Storage | `GSClient`
`GSPath` | `LocalGSClient`
`LocalGSPath` | `local_gs_implementation` |\n", - "| Amazon S3 | `S3Client`
`S3Path` | `LocalS3Client`
`LocalS3Path` | `local_s3_implementation` |\n" + "| Amazon S3 | `S3Client`
`S3Path` | `LocalS3Client`
`LocalS3Path` | `local_s3_implementation` |\n", + "\n", + "The `Local*` path classes also support presigned URLs. Calling `as_url(presign=True)` returns a deterministic stand-in URL with `expires` and `signature` query parameters (for example, `s3://mybucket/myfile.txt?expires=3600&signature=local`) instead of raising an error, so code that hands out presigned URLs can be exercised in your tests. The URL is not actually signed and can't be fetched; assert on its parts rather than trying to use it.\n" ] }, { diff --git a/pyproject.toml b/pyproject.toml index 6cf6995a..63ec39f6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "flit_core.buildapi" [project] name = "cloudpathlib" -version = "0.24.0" +version = "0.25.0" description = "pathlib-style classes for cloud storage services." readme = "README.md" authors = [{ name = "DrivenData", email = "info@drivendata.org" }]