Skip to content

Replace hand-written DMS client with the Go SDK - #6591

Draft
shreyas-goenka wants to merge 1 commit into
mainfrom
dms-sdk-client
Draft

Replace hand-written DMS client with the Go SDK#6591
shreyas-goenka wants to merge 1 commit into
mainfrom
dms-sdk-client

Conversation

@shreyas-goenka

Copy link
Copy Markdown
Contributor

The libs/dms rawClient hand-wrote CreateVersion, UpdateDeployment, and UpdateOperation because the SDK lacked the operations field, the method, and int64 sequence_id handling. All three landed in SDK v0.177.0, so this routes them through the generated client, using ForceSendFields to keep DMS's masked-but-empty fields (a cleared deployment_mode, an empty error_message, a staged sequence_id of 0) on the wire. The testserver fake now reads sequence_id as the number the SDK sends and still emits it as a string, like the real service.

This pull request and its description were written by Isaac.

The libs/dms rawClient hand-wrote CreateVersion, UpdateDeployment, and
UpdateOperation because the SDK lacked the operations field, the method, and
int64 sequence_id handling. All three are in SDK v0.177.0, so route them through
the generated client, using ForceSendFields to keep DMS's masked-but-empty fields
on the wire. The testserver fake now reads sequence_id as the number the SDK sends
(still emitting a string, like the real service).

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 96b355e

Run: 34374353395

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
🔄​ aws linux 6 1 288 21 19:46
🔄​ aws windows 3 294 19 14:29
🔄​ azure linux 1 1 292 21 15:08
🔄​ azure windows 6 1 289 19 19:43
🔄​ gcp linux 1 1 293 21 19:45
🔄​ gcp windows 8 1 288 19 21:13
20 interesting tests: 20 flaky
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🔄​ TestAccept 💚​R 🔄​f 💚​R 💚​R 💚​R 💚​R
🔄​ TestAccept/bundle/dms/declined-deploy 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/dms/declined-deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/dms/depends-on ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/dms/depends-on/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/dms/deployment-metadata-change ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/dms/multiple-resources ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/dms/multiple-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/dms/provenance ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/dms/provenance/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/dms/record-failure 🔄​f ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/dms/record-failure/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true 🔄​f ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/dms/stale-plan 🔄​f 🔄​f ✅​p 🔄​f ✅​p ✅​p
🔄​ TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN= 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1 ✅​p 🔄​f ✅​p 🔄​f ✅​p ✅​p
🔄​ TestSyncFullFileSync ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestSyncIncrementalFileSync ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
🔄​ TestSyncIncrementalSyncFileToPythonNotebook ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p
🔄​ TestSyncNestedFolderDoesntFailOnNonEmptyDirectory ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
Top 44 slowest tests (at least 2 minutes):
duration env testname
5:42 azure windows TestFilerWorkspaceFilesExtensionsReadDir
4:10 azure windows TestFilerWorkspaceFilesExtensionsStat
3:48 azure windows TestAccept/bundle/dms/declined-deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:46 azure linux TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:39 gcp linux TestFilerReadWrite/workspace_files
3:38 gcp windows TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:38 gcp linux TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:32 aws windows TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=
3:31 gcp linux TestFilerWorkspaceFilesExtensionsDelete
3:27 gcp linux TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=
3:22 gcp linux TestAccept
3:18 gcp linux TestAccept/bundle/dms/depends-on/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:16 aws windows TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1
3:07 aws linux TestFilerWorkspaceFilesExtensionsReadDir
3:04 azure windows TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
3:03 aws linux TestFilerWorkspaceNotebook/scalaJupyterNb.ipynb
3:01 gcp windows TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1
2:57 aws linux TestAccept/bundle/dms/multiple-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:55 aws windows TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:51 aws linux TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:50 gcp linux TestFilerWorkspaceFilesExtensionsReadDir
2:44 gcp windows TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:40 gcp windows TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:40 azure windows TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:37 azure linux TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:36 gcp linux TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1
2:32 aws windows TestFilerRecursiveDelete/workspace_files_extensions
2:28 azure windows TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=
2:26 azure linux TestImportDirWithOverwriteFlag
2:25 azure windows TestFilerReadDir/workspace_files
2:25 azure linux TestAccept
2:18 azure windows TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1
2:17 azure windows TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
2:13 aws windows TestFilerRecursiveDelete/workspace_files
2:10 gcp linux TestFilerWorkspaceFilesExtensionsRead
2:09 azure linux TestFilerWorkspaceFilesExtensionsStat
2:06 aws windows TestAccept/bundle/dms/multiple-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:06 aws windows TestFilerWorkspaceFilesExtensionsStat
2:05 gcp windows TestFilerWorkspaceFilesExtensionsReadDir
2:04 gcp linux TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:02 aws linux TestFilerRecursiveDelete/workspace_files
2:02 azure windows TestAccept/bundle/dms/provenance/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true
2:02 aws linux TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=
2:00 azure linux TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants