Skip to content

feat: add Google Cloud Storage backend - #1

Open
dylayed wants to merge 1 commit into
storage-backend-refactorfrom
gcs-support-review
Open

feat: add Google Cloud Storage backend#1
dylayed wants to merge 1 commit into
storage-backend-refactorfrom
gcs-support-review

Conversation

@dylayed

@dylayed dylayed commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Summary

This adds Google Cloud Storage as an alternative backend on top of #2. An exact gs://BUCKET target selects GCS; existing bare bucket names and s3://BUCKET continue to select S3.

GCS uses Application Default Credentials and object generations for conditional writes. Deployment uploads, runtime bucket access, ownership, leases, and LTX replication all use the selected backend.

Approach

  • Validate and normalize the storage target before control-plane enrollment. S3 keeps its existing enrollment form; GCS uses gs://BUCKET with no endpoint or region identity.
  • Build GCS with the standard ADC file override, local ADC, or metadata identity. Custom GCS endpoints and managed GCP credentials are not included.
  • Map GCS generations to the same opaque version value used by the ownership code. Conditional-write retries remain disabled.
  • Run the existing celld-ltx object-store conformance suite against GCS beside the MinIO integration test.

This is intentionally a narrow first integration: no bucket prefixes, emulator support, or cross-provider migration.

Review

The main things to review are provider selection and enrollment normalization, ADC setup, and generation-based CAS in crates/celld/storage_backend.rs. Most other changes make existing output and documentation name the selected provider correctly.

Review #2 first. The two PRs are linked in native stack #3.

Validation

  • Celld checks and the celld-ltx S3/GCS test build pass.
  • Focused provider, enrollment, and ownership tests pass.
  • Real MinIO regression coverage passes the CAS contract and all 3 LTX integration tests, including multipart.
  • Live GCS conformance completed without a skip and cleanup left no objects behind.
  • A two-worker GCS deployment preserved counter writes and restored state across worker replacement.

@dylayed
dylayed force-pushed the gcs-support-review branch from 81f48f8 to 001fe73 Compare August 7, 2026 16:22
@dylayed dylayed changed the title feat: support Google Cloud Storage backend feat: add Google Cloud Storage backend Aug 7, 2026
@dylayed
dylayed changed the base branch from main to storage-backend-refactor August 7, 2026 16:23
@dylayed
dylayed force-pushed the gcs-support-review branch from 001fe73 to ac214ea Compare August 7, 2026 17:24
@dylayed
dylayed force-pushed the gcs-support-review branch from ac214ea to 365b403 Compare August 7, 2026 17:58
@dylayed
dylayed force-pushed the gcs-support-review branch from 365b403 to 0a384c1 Compare August 7, 2026 19:40
@dylayed
dylayed force-pushed the gcs-support-review branch from 0a384c1 to eca158b Compare August 8, 2026 01:08
@dylayed
dylayed force-pushed the gcs-support-review branch 2 times, most recently from 4d9c858 to 4c78306 Compare August 8, 2026 02:32
@dylayed
dylayed force-pushed the gcs-support-review branch 4 times, most recently from aeec7fc to f82b778 Compare August 8, 2026 03:35
@dylayed
dylayed force-pushed the gcs-support-review branch from f82b778 to 63298c8 Compare August 8, 2026 03:44
@dylayed
dylayed force-pushed the gcs-support-review branch 2 times, most recently from 0df5c6a to a74334d Compare August 8, 2026 13:51
@dylayed
dylayed force-pushed the gcs-support-review branch 3 times, most recently from 8e9fd78 to 51298ad Compare August 8, 2026 14:24
@dylayed
dylayed force-pushed the gcs-support-review branch 2 times, most recently from b22f13d to 7ab4b19 Compare August 8, 2026 20:17
@dylayed
dylayed force-pushed the gcs-support-review branch 8 times, most recently from a3e72b4 to 54c8f18 Compare August 9, 2026 16:17
@dylayed
dylayed force-pushed the gcs-support-review branch 2 times, most recently from 4307511 to bcd0ea5 Compare August 9, 2026 17:06
@dylayed
dylayed force-pushed the gcs-support-review branch from bcd0ea5 to e975f0f Compare August 9, 2026 22:39
@dylayed
dylayed force-pushed the gcs-support-review branch from e975f0f to 2f93322 Compare August 10, 2026 01:25
@dylayed
dylayed force-pushed the gcs-support-review branch from 2f93322 to 3df26bf Compare August 10, 2026 01:37
@dylayed
dylayed force-pushed the gcs-support-review branch 3 times, most recently from 62d2a5e to af5f9ce Compare August 10, 2026 17:18
@dylayed
dylayed force-pushed the gcs-support-review branch from af5f9ce to ec57467 Compare August 10, 2026 18:23
Select GCS explicitly with gs:// bucket URIs, authenticate through Google Application Default Credentials, and translate object generations into the shared conditional-write version contract.
@dylayed
dylayed force-pushed the gcs-support-review branch from ec57467 to 935b3d4 Compare August 10, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant