feat(store): implement pluggable cloud storage for HStore

[vaijosh]

Purpose

• Closes: [Feature] Implement Cloud Storage Support for HugeGraph #3079

This PR introduces a cloud storage architecture for HugeGraph Store (HStore). In cloud-native environments, storage nodes are ephemeral, making reliance on local disk storage a single point of failure. This implementation decouples the storage layer from local disk dependencies, allowing SST files to be offloaded to durable, scalable cloud storage providers.

Key Changes

Pluggable Architecture

• Created the CloudStorageProvider SPI interface to enable extensible storage backends.
• Added CloudStorageProviderFactory for seamless provider discovery and lifecycle management.

S3 Provider Implementation

• Introduced the hg-store-cloud-s3 module, leveraging AWS SDK v2 for production-ready S3/S3 compatible storage interaction.

Lifecycle & Event Integration

• Integrated CloudStorageEventListener with RocksDBFactory. This hooks into critical SST lifecycle events (onTableFileCreated, onTableFileDeleted) to ensure synchronized state between local RocksDB and cloud storage.
• Implemented startup hydration (onDBCreated) to backfill pre-existing files and read-miss on-demand hydration to ensure data availability.

Infrastructure & Testing

• Added a comprehensive local development environment via docker/cloud-storage/ using MinIO.
• Included an integration test script (test-graph-queries-and-sst.sh) to verify end-to-end data durability and query consistency.

Implementation Highlights

• SPI Integration: Standardized service discovery via META-INF/services.
• RocksDB Hooking: Registered a singleton RocksdbEventListener within RocksDBFactory to intercept file operations without modifying core RocksDB logic.
• Read-Miss Protection: Added a guard window mechanism in onReadMiss to prevent redundant hydration requests during high-concurrency read scenarios.

Verifying These Changes

• Unit Tests: Coverage for config parsing (CloudStorageConfigTest), factory registration, and event listener logic.
• Integration Tests: Verified using the new docker-compose setup with MinIO.
• Configuration: Verified application.yml bindings for credentials, bucket management, and sync intervals.

Impact

• Dependencies: Added AWS SDK v2; updated LICENSE and NOTICE accordingly.
• Configurations: Added cloud.storage namespace to application.yml (disabled by default).
• Public API: No breaking changes to existing public-facing client APIs.

Documentation

• Architecture and configuration details are documented in hugegraph-store/docs/pluggable-cloud-storage-architecture.md.

[codecov]

Codecov Report

❌ Patch coverage is 95.95960% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.20%. Comparing base (4f1a8b3) to head (7823d3f).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...he/hugegraph/store/metric/SystemMetricService.java	0.00%	3 Missing ⚠️
...ache/hugegraph/store/cloud/CloudStorageConfig.java	93.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3081      +/-   ##
============================================
- Coverage     39.30%   36.20%   -3.11%     
- Complexity      264      463     +199     
============================================
  Files           771      775       +4     
  Lines         65915    66036     +121     
  Branches       8754     8771      +17     
============================================
- Hits          25910    23908    -2002     
- Misses        37241    39474    +2233     
+ Partials       2764     2654     -110     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[VGalaxies]

Review summary

• Blocking: yes
• Summary: The change cannot currently guarantee recoverable RocksDB state, may silently disable cloud storage, and contains S3 retry and deployment defects.
• Evidence:
  • Static analysis of git diff origin/master...HEAD
  • shell syntax and POM XML checks passed
  • git diff --check reported whitespace-only errors

[VGalaxies]

🚨 Review summary

Important

The cloud recovery path has blocking consistency, configuration, deletion-safety, release-metadata, and verification gaps.

📊 Risk dashboard

Signal	Result
🚦 Review gate	Blocked
🔎 Actionable findings	9 (8 High, 1 Medium)
🧪 Verification coverage	5 checks
⚠️ Confirmed / limited checks	2 / 1

🔬 Coverage details

⚠️ S3 failure semantics — Confirmed

Trace: S3CloudStorageProvider upload, download, delete, and exception-classification paths, CloudStorageEventListener purge and hydration flows, AWS SDK 2.33.8 retry-code and file-transformer behavior

Conclusion: Confirmed ignored per-object deletion errors, retryable error-code misclassification, and non-atomic download recovery risks; lower-impact lifecycle leads were dropped.

⚠️ Release and E2E integrity — Confirmed

Trace: install-dist release LICENSE and known-dependencies inventory, cloud-storage Docker artifact build and recovery workflow, Maven provider packaging and test activation

Conclusion: Confirmed the release dependency inventory mismatch and E2E false-green paths caused by suppressed build and wipe failures.

🟡 Focused unit suites — Limited

Trace: hg-store-common cloud configuration, provider, and factory tests, hg-store-cloud-s3 exception-classification tests, hg-store-node listener, retry, metrics, tracker, configuration, and callback tests, hg-store-core BusinessHandlerImplTest

Conclusion: Surefire reports recorded 184 tests with 0 failures, 0 errors, and 0 skipped. The suites do not cover concurrent metadata publication, comma-separated data roots, or partial S3 batch deletion.

✅ Maven reactor — Clear

Trace: Full 44-project reactor with the cloud-s3 profile, hugegraph-store module ordering and dependency graph

Conclusion: Maven validate completed successfully for all 44 reactor projects; the suspected duplicate cloud-s3 module activation did not reproduce.

✅ Shell syntax — Clear

Trace: pd-entrypoint.sh, store-entrypoint.sh, test-graph-queries-and-sst.sh

Conclusion: bash -n returned successfully for all three changed shell entrypoints and workflows.

Warning

Verification limits

• No live Docker and MinIO recovery drill was run, so cluster recovery and purge behavior were not exercised end to end.
• The test environment used Java 17; the legacy JaCoCo agent emitted unsupported class-version instrumentation errors, so coverage instrumentation was not verified even though test assertions passed.
• S3SingleLargeFileE2ETest was intentionally not run because its default path generates a 20 GiB file.
• The independent recovery-consistency run timed out after 900 seconds and produced no usable result.

---

_{🤖 Codex review · GPT-5.6 Sol · effort: xhigh}

[vaijosh]

🚨 Review summary

Important

The cloud recovery path has blocking consistency, configuration, deletion-safety, release-metadata, and verification gaps.

📊 Risk dashboard

Signal Result
🚦 Review gate Blocked
🔎 Actionable findings 9 (8 High, 1 Medium)
🧪 Verification coverage 5 checks
⚠️ Confirmed / limited checks 2 / 1

🔬 Coverage details

⚠️ S3 failure semantics — Confirmed
Trace: S3CloudStorageProvider upload, download, delete, and exception-classification paths, CloudStorageEventListener purge and hydration flows, AWS SDK 2.33.8 retry-code and file-transformer behavior

Conclusion: Confirmed ignored per-object deletion errors, retryable error-code misclassification, and non-atomic download recovery risks; lower-impact lifecycle leads were dropped.

⚠️ Release and E2E integrity — Confirmed
Trace: install-dist release LICENSE and known-dependencies inventory, cloud-storage Docker artifact build and recovery workflow, Maven provider packaging and test activation

Conclusion: Confirmed the release dependency inventory mismatch and E2E false-green paths caused by suppressed build and wipe failures.

🟡 Focused unit suites — Limited
Trace: hg-store-common cloud configuration, provider, and factory tests, hg-store-cloud-s3 exception-classification tests, hg-store-node listener, retry, metrics, tracker, configuration, and callback tests, hg-store-core BusinessHandlerImplTest

Conclusion: Surefire reports recorded 184 tests with 0 failures, 0 errors, and 0 skipped. The suites do not cover concurrent metadata publication, comma-separated data roots, or partial S3 batch deletion.

✅ Maven reactor — Clear
Trace: Full 44-project reactor with the cloud-s3 profile, hugegraph-store module ordering and dependency graph

Conclusion: Maven validate completed successfully for all 44 reactor projects; the suspected duplicate cloud-s3 module activation did not reproduce.

✅ Shell syntax — Clear
Trace: pd-entrypoint.sh, store-entrypoint.sh, test-graph-queries-and-sst.sh

Conclusion: bash -n returned successfully for all three changed shell entrypoints and workflows.

Warning

Verification limits

• No live Docker and MinIO recovery drill was run, so cluster recovery and purge behavior were not exercised end to end.
• The test environment used Java 17; the legacy JaCoCo agent emitted unsupported class-version instrumentation errors, so coverage instrumentation was not verified even though test assertions passed.
• S3SingleLargeFileE2ETest was intentionally not run because its default path generates a 20 GiB file.
• The independent recovery-consistency run timed out after 900 seconds and produced no usable result.

🤖 Codex review · GPT-5.6 Sol · effort: xhigh

Hi @VGalaxies.
Thanks for reviewing the changes. I have addressed all the review comments. Can you please review it again?

[vaijosh]

@VGalaxies
I am not sure why code coverage check is failing.
We have
The Diff has 94.79% coverage (agains target 42.32%) [codecov/patch](https://github.com/apache/hugegraph/pull/3081/checks?check_run_id=89150435397)Successful in 1s — 94.79% of diff hit (target 42.32%)

Not sure why the coverage is suddenly dropped at project level to 30.60% (-11.72%) compared to e960cc5 codecov/projectFailing after 1s — 30.60% (-11.72%) compared to e960cc5

[Copilot]

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/business/BusinessHandlerImpl.java:1042

• The new truncate() block is not indented like the rest of the file (4-space indent), which makes the method hard to read and diverges from surrounding code style. Please re-indent the try-with-resources and its inner blocks consistently.

try (RocksDBSession dbSession = getSession(graphName, partId)) {
    String dbName = dbSession.getGraphName();
    String dbPath = dbSession.getDbPath();
    factory.notifyTruncateBegin(dbName, dbPath);
    boolean truncated = false;
    try {
        dbSession.sessionOp().deleteRange(keyCreator.getStartKey(partId, graphName),
                                          keyCreator.getEndKey(partId, graphName));
        // Release map ID
        keyCreator.delGraphId(partId, graphName);
        truncated = true;
    } finally {
        if (truncated) {
            factory.notifyTruncate(dbName, dbPath);
        } else {
            factory.notifyTruncateAbort(dbName, dbPath);
        }
    }
}
    }

hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageConfig.java:156

• The Javadoc for providerProperties ends mid-sentence (trailing comma), which makes the public configuration docs look incomplete. Please finish the sentence and clarify what belongs in this map.

    /**
     * Provider-specific properties, mapped from {@code cloud.storage.<provider>.*} in the configuration.
     * These are passed verbatim to the provider implementation and may include credentials,
     */
    private Map<String, String> providerProperties = new HashMap<>();

[Copilot]

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (9)

hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/SessionOperatorReadMissTest.java:1

• The test creates a RocksDBSession but never closes it, relying on factory destroy/cleanup. This can leak native RocksDB resources and can make temp-dir deletion flaky (especially on Windows). Use try-with-resources for session (or explicitly session.close() before destroyGraphDB).
  hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBFactoryTest.java:1
• Like the read-miss test, this test creates a RocksDBSession that is not closed on the success path. Please wrap the session in try-with-resources (or close it in finally) to avoid leaking native handles and to reduce flakiness when deleting the temp directory.
  hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsConst.java:1
• This metric is registered as a Gauge in CloudStorageMetrics, but the name suffix _total is conventionally reserved for Counters in Prometheus/Micrometer ecosystems. Consider renaming it to something gauge-appropriate (e.g., _confirmed_files or _confirmed_files_count) to avoid confusion and incorrect dashboard assumptions.
  hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetrics.java:1
• Catching IllegalArgumentException as a control-flow mechanism is brittle here: Micrometer throws IllegalArgumentException for multiple reasons (not only 'already registered'), and a partial registration could leave perDbMeters missing/incorrect (making unregisterDatabaseMetrics() unable to remove all meters). A safer approach is to guard via perDbMeters.containsKey(dbName) / computeIfAbsent and only register once per dbName, and only swallow the specific duplicate-registration scenario if you can reliably detect it.
  hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetrics.java:1
• Catching IllegalArgumentException as a control-flow mechanism is brittle here: Micrometer throws IllegalArgumentException for multiple reasons (not only 'already registered'), and a partial registration could leave perDbMeters missing/incorrect (making unregisterDatabaseMetrics() unable to remove all meters). A safer approach is to guard via perDbMeters.containsKey(dbName) / computeIfAbsent and only register once per dbName, and only swallow the specific duplicate-registration scenario if you can reliably detect it.
  hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/AppConfigCloudStorageTest.java:1
• The last assertion is ineffective: readProviderProperties() strips the cloud.storage.<provider>. prefix, so the map will never contain a key like cloud.storage.s3.bucket regardless of implementation correctness. To validate isolation, assert something observable in the returned map (e.g., that bucket equals gcs-bucket and not s3-bucket, and/or that only the expected gcs keys are present).
  hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableExceptionTest.java:58
• This test currently passes even if the serialVersionUID field is missing or inaccessible because the catch block swallows the failure. The catch should fail the test (e.g., via fail(...)) so that regressions in serialization metadata are actually detected.

     public void testSerialVersionUID() {
         try {
             java.lang.reflect.Field field =
                     CloudStorageNonRetryableException.class.getDeclaredField("serialVersionUID");
             field.setAccessible(true);
             Long serialVersionUID = (Long) field.get(null);
             assertEquals(Long.valueOf(1L), serialVersionUID);
         } catch (NoSuchFieldException | IllegalAccessException e) {
             // serialVersionUID field exists as expected
         }
     }

hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageConfig.java:157

• Since providerProperties is a mutable Map, API consumers can accidentally share and mutate it across components (and tests in this PR even rely on aliasing). For safer config handling, consider defensively copying the map in the setter/getter (or returning an unmodifiable view) so provider configuration is stable after initialization.

    /**
     * Provider-specific properties, mapped from {@code cloud.storage.<provider>.*} in the
     * configuration. These are passed verbatim to the provider implementation and may include
     * credentials, endpoint overrides, regions, or SDK tuning keys expected by that provider.
     */
    private Map<String, String> providerProperties = new HashMap<>();

docker/cloud-storage/docker-compose.yml:30

• Using minio/minio:latest makes the dev/integration environment non-reproducible and can break unexpectedly when MinIO releases change behavior. Pin to a specific MinIO version tag (similar to how minio/mc is pinned below) to keep CI/dev validation deterministic.

  minio:
    image: minio/minio:latest

[Copilot]

Pull request overview

Copilot reviewed 57 out of 59 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (6)

hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBFactoryTest.java:1

• This test hard-asserts an implementation detail of RocksDB checkpoints (that they flush memtables and emit SST-created events), and uses time-based waiting + System.out.printf, which can make CI runs flaky across platforms/RocksDB versions. Consider making the test assert only the project-level contract you need (e.g., that your chosen metadata-sync path does/doesn’t trigger file-created callbacks under your own code), or convert this into a non-failing diagnostic/assumption-style probe and remove stdout printing (use logging if needed).
  hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBFactoryTest.java:1
• This test hard-asserts an implementation detail of RocksDB checkpoints (that they flush memtables and emit SST-created events), and uses time-based waiting + System.out.printf, which can make CI runs flaky across platforms/RocksDB versions. Consider making the test assert only the project-level contract you need (e.g., that your chosen metadata-sync path does/doesn’t trigger file-created callbacks under your own code), or convert this into a non-failing diagnostic/assumption-style probe and remove stdout printing (use logging if needed).
  hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBFactoryTest.java:1
• This test hard-asserts an implementation detail of RocksDB checkpoints (that they flush memtables and emit SST-created events), and uses time-based waiting + System.out.printf, which can make CI runs flaky across platforms/RocksDB versions. Consider making the test assert only the project-level contract you need (e.g., that your chosen metadata-sync path does/doesn’t trigger file-created callbacks under your own code), or convert this into a non-failing diagnostic/assumption-style probe and remove stdout printing (use logging if needed).
  hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/SessionOperatorReadMissTest.java:1
• RocksDBSession is never explicitly closed here. Relying on destroyGraphDB() for cleanup can leak native resources if the session remains referenced or if destroy fails, and it can also cause temp directory deletion failures on some platforms. Consider using try-with-resources for RocksDBSession (or closing it in the finally block before destroy) to make the test cleanup deterministic.
  hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/metrics/CloudStorageMetricsTest.java:1
• The static reset helper clears several fields but does not reset the *_supplier static fields (retryQueueSizeSupplier, dlqPersistenceHealthySupplier, deleteMarkerHealthySupplier). Because those suppliers can capture references to prior test objects, leaving them untouched can introduce hidden inter-test coupling and memory leaks. Consider explicitly resetting them to their documented defaults (0/1) as part of resetStaticState().
  docker/cloud-storage/docker-compose.yml:31
• Using minio/minio:latest makes the local integration environment non-reproducible and can cause sudden CI/dev breakages when MinIO changes behavior. Pin to a specific MinIO release tag (similar to how minio/mc is pinned) so recovery/hydration tests remain stable over time.

    image: minio/minio:latest
    container_name: cloud-storage-minio

[Copilot]

Pull request overview

Copilot reviewed 58 out of 60 changed files in this pull request and generated no new comments.

Suppressed comments (1)

hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBSessionTest.java:24

• This test file now has unused imports (e.g., static import fail, java.nio.file.Files/Path, and org.junit.Test). In Java, unused imports fail compilation, and this class currently has no active @Test methods (the annotations are commented out), so these imports should be removed (or the tests re-enabled).

[vaijosh]

Pull request overview

Copilot reviewed 58 out of 60 changed files in this pull request and generated no new comments.

Suppressed comments (1)
hugegraph-store/hg-store-rocksdb/src/test/java/org/apache/hugegraph/rocksdb/access/RocksDBSessionTest.java:24

• This test file now has unused imports (e.g., static import fail, java.nio.file.Files/Path, and org.junit.Test). In Java, unused imports fail compilation, and this class currently has no active @Test methods (the annotations are commented out), so these imports should be removed (or the tests re-enabled).

Implemented this review comment. Removed unused imports.

[vaijosh]

Hi @VGalaxies @imbajin
Please don't merge the PR.
I can see that JDK17 changes are in master branch. So I will need to test my changes with JRE17. I will update here once my testing is done.

[vaijosh]

Hi @VGalaxies @imbajin Please don't merge the PR. I can see that JDK17 changes are in master branch. So I will need to test my changes with JRE17. I will update here once my testing is done.

Please ignore. I was bit confused due to merge conflict I got in docker/README.md due to changes in 9aaad11. It seems only reference to JDK17 was removed from README.md. But we are not yet moved to JDK17.

[vaijosh]

@imbajin @VGalaxies
I have committed few more changes, fixed multiple review comments by Claude Code and Github Copilot.

Tested following scenarios

1. SST and metadata synching is happening properly when Flush happens - End to End test and Manual steps in README.md
2. During Compaction the new files are synced and few files are deleted. - End to End tests.
3. During disk failures, the rehydration happens via Cloud storage. - End to End test and Manual steps in README.md
4. During delete scenarios the TOMBSTONE is written and when delete is complete the data is deleted. - End to End test
5. Load IBM AML large data ( Around 677609 Vertices and 17843793 Edgest) - aml_sql_showcase_hugegraph.ipynb