HDDS-14108. Provide option in ‘scm safemode status’ to show status of all SCM nodes by sreejasahithi · Pull Request #9611 · apache/ozone

sreejasahithi · 2026-01-09T07:07:45Z

What changes were proposed in this pull request?

This PR provides an option --all to show the safemode status of each SCM node in the cluster.
If verbose, It also provides the status of each safemode exit rule for each SCM node.

This PR also fixes the bug stated in HDDS-13832 where when --scm option is used in HA it always shows the status of the leader SCM and silently ignores the node specified via the option.

What is the link to the Apache JIRA

HDDS-14108

How was this patch tested?

This patch was tested locally in a docker ozone-ha cluster:

bash-5.1$ ozone admin safemode status --all
Service ID: scmservice
scm1:9860 [scm1]: out of safe mode
scm2:9860 [scm2]: out of safe mode
scm3:9860 [scm3]: out of safe mode
bash-5.1$ ozone admin safemode status --all --verbose
Service ID: scmservice
scm1:9860 [scm1]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
scm2:9860 [scm2]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
scm3:9860 [scm3]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)

bash-5.1$ ozone admin safemode status --scm=scm2:9860
Service ID: scmservice
scm2:9860 [scm2]: out of safe mode
bash-5.1$ ozone admin safemode status --scm=scm2:9860 --verbose
Service ID: scmservice
scm2:9860 [scm2]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)

bash-5.1$ ozone admin safemode status      
Service ID: scmservice
scm1:9860 [scm1]: out of safe mode
bash-5.1$ ozone admin safemode status --verbose
Service ID: scmservice
scm1:9860 [scm1]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)

if scm1 goes down then we can see that the command curretly picks up the new leader when we run the command with no explicit option (--all or --scm) provided.

bash-5.1$ ozone admin safemode status
Service ID: scmservice
scm2:9860 [scm2]: out of safe mode
bash-5.1$ ozone admin safemode status --verbose
Service ID: scmservice
scm2:9860 [scm2]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
bash-5.1$  ozone admin scm roles
scm1:9894:FOLLOWER:bae9a4cf-ce85-47f3-91d3-f90fa1989e2e:
scm3:9894:FOLLOWER:04822469-9758-4429-9aec-b38c5c707882:172.18.0.6
scm2:9894:LEADER:9cda6813-c047-4af5-b1f6-25b6a3ac7bfa:172.18.0.11

bash-5.1$ ozone admin safemode status --all
Service ID: scmservice
scm1:9860 [scm1]: ERROR: Failed to get safe mode status for SCM node: Invalid host name: local host is: "om1/172.18.0.2"; destination host is: "scm1":9860; java.net.UnknownHostException: Invalid host name: local host is: "om1/172.18.0.2"; destination host is: "scm1":9860; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
scm2:9860 [scm2]: out of safe mode
scm3:9860 [scm3]: out of safe mode
bash-5.1$ ozone admin safemode status --all --verbose
Service ID: scmservice
scm1:9860 [scm1]: ERROR: Failed to get safe mode status for SCM node: Invalid host name: local host is: "om1/172.18.0.2"; destination host is: "scm1":9860; java.net.UnknownHostException: Invalid host name: local host is: "om1/172.18.0.2"; destination host is: "scm1":9860; java.net.UnknownHostException; For more details see:  http://wiki.apache.org/hadoop/UnknownHost; For more details see:  http://wiki.apache.org/hadoop/UnknownHost
scm2:9860 [scm2]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
scm3:9860 [scm3]: out of safe mode
validated:true, DataNodeSafeModeRule, registered datanodes (=3) >= required datanodes (=3)
validated:true, RatisContainerSafeModeRule, 100.00% of [RATIS] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)
validated:true, HealthyPipelineSafeModeRule, healthy Ratis/THREE pipelines (=1) >= healthyPipelineThresholdCount (=1)
validated:true, StateMachineReadyRule, Refreshed SCM State Machine after leader ready: true
validated:true, OneReplicaPipelineSafeModeRule, reported Ratis/THREE pipelines with at least one datanode (=0) >= threshold (=0)
validated:true, ECContainerSafeModeRule, 100.00% of [EC] Containers(0 / 0) with at least N reported replica (=1.00) >= safeModeCutoff (=0.99)

if scm node scm2 also goes down then it clearly states that leader could not be determined.

bash-5.1$ ozone admin safemode status
Service ID: scmservice
Could not determine leader node

when invalid scm node provided:

bash-5.1$ ozone admin safemode status --scm=scm10:9860          
Service ID: scmservice
Specified --scm address scm10:9860 does not match any node in service scmservice. Nodes: scm1:9860 [scm1], scm2:9860 [scm2], scm3:9860 [scm3]

Green CI : https://github.com/sreejasahithi/ozone/actions/runs/20842284515

… all SCM nodes

dombizita · 2026-01-10T10:36:35Z

@octachoron would you like to take a look at it if you have time? It's related to what we discussed recently :)

octachoron · 2026-01-11T03:26:17Z

@dombizita, absolutely, thank you! I don't think my vote is enough to merge though.

octachoron

Thank you @sreejasahithi for the patch. I added a few thoughts and questions inline. 🙂

octachoron

Thank you, the changes look good to me. Do you think there is a good way to write tests for the feature? (I do not see straightforward precedent other than actual integration tests, but that does not mean there isn't a way. 🙂)

ashishkumar50

@sreejasahithi Thanks for working on this.

ashishkumar50 · 2026-01-13T11:13:45Z

+  }
+
+  private void executeForSpecificNodeInHA(ScmClient scmClient, String serviceId) throws IOException {
+    String scmAddress = getScmOption().getScm();


scmAddress is not mandatory option.

ashishkumar50 · 2026-01-13T11:16:13Z

+    } else if (StringUtils.isNotEmpty(getScmOption().getScm()) && serviceId != null) {
+      executeForSpecificNodeInHA(scmClient, serviceId);
+    } else {
+      executeForSingleNode(scmClient);


In normal or existing behaviour we need safemode status from leader node most of the time. When no scm address is passed, whether we are getting safe mode status from leader node or not? Because now follower also can accept safemode and can return the status.

Thanks @ashishkumar50 for finding this bug, you are right now that we are allowing follower to also accept status command there can be a possibility where when we run safemode status command with no additional option it can return the status of the follower.

I have fixed this issue.

priyeshkaratha

Thanks @sreejasahithi for working on this. I have one minor comment on handling audit logs.

sadanand48 · 2026-01-14T08:08:55Z

+          target.getAddress().equals(nodeAddr.getAddress());
+    } catch (Exception e) {
+      // If address resolution fails, no match
+      return false;


nit : Log the exception here before returning false

I have removed the logging here because it creates unwanted noise in the CLI output. Instead, I have ensured that actual errors are properly surfaced when no leader can be determined or when the node specified in --scm option doesn't match, clear error messages are thrown to the user.

priyeshkaratha

Thanks @sreejasahithi for improving the PR. Changes looks good to me.

aryangupta1998

Thanks for the patch @sreejasahithi, some minor comments inline!

aryangupta1998

Thanks for updating the patch @sreejasahithi, LGTM!

sreejasahithi · 2026-02-11T04:57:35Z

@sumitagrawl could you please review this patch.

sumitagrawl

LGTM

szetszwo · 2026-08-15T20:36:05Z

  public Map<String, Pair<Boolean, String>> getSafeModeRuleStatuses()
      throws IOException {
    GetSafeModeRuleStatusesRequestProto request =
        GetSafeModeRuleStatusesRequestProto.getDefaultInstance();
    GetSafeModeRuleStatusesResponseProto response =
        submitRequest(Type.GetSafeModeRuleStatuses,
            builder -> builder.setGetSafeModeRuleStatusesRequest(request))
            .getGetSafeModeRuleStatusesResponse();
-    Map<String, Pair<Boolean, String>> map = new HashMap();
-    for (SafeModeRuleStatusProto statusProto :
-        response.getSafeModeRuleStatusesProtoList()) {
-      map.put(statusProto.getRuleName(),
+    return buildSafeModeRuleStatusesMap(response);
+  }
+
+  /**
+   * Helper method to build a map from GetSafeModeRuleStatusesResponseProto.
+   * Extracts rule names and their status information.
+   */
+  private Map<String, Pair<Boolean, String>> buildSafeModeRuleStatusesMap(
+      GetSafeModeRuleStatusesResponseProto response) {
+    Map<String, Pair<Boolean, String>> ruleStatuses = new HashMap<>();
+    for (SafeModeRuleStatusProto statusProto : response.getSafeModeRuleStatusesProtoList()) {
+      ruleStatuses.put(statusProto.getRuleName(),
          Pair.of(statusProto.getValidate(), statusProto.getStatusText()));
    }
-    return map;
+    return ruleStatuses;
  }


@sreejasahithi , Using Map + Pair is an incorrect choice of data structure. Just want to let you know but not blaming your since it was the existing code before this PR. See

HDDS-16130. Remove Pair usage from Safe-mode APIs and callers. #10993

https://issues.apache.org/jira/secure/attachment/13083934/10993_review.patch

Thanks @szetszwo for pointing this out. Agreed Map<String, Pair<Boolean, String>> is not needed given the use of SafeModeRuleStatusProto.
Will take a look at HDDS-16130

HDDS-14108. Provide option in ‘scm safemode status’ to show status of…

14173ff

… all SCM nodes

dombizita self-requested a review January 10, 2026 10:35

octachoron reviewed Jan 11, 2026

View reviewed changes

Refactored code for better readability and reusability

5958634

sreejasahithi requested a review from octachoron January 12, 2026 10:27

jojochuang requested review from errose28 and sumitagrawl and removed request for octachoron January 12, 2026 17:27

octachoron reviewed Jan 13, 2026

View reviewed changes

ashishkumar50 reviewed Jan 13, 2026

View reviewed changes

priyeshkaratha reviewed Jan 14, 2026

View reviewed changes

Comment thread ...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java Outdated

sadanand48 reviewed Jan 14, 2026

View reviewed changes

Sreeja Chintalapati added 2 commits January 16, 2026 15:33

Should return leader node status when no option provided

4b95b95

Addressed review comments wrt logs

dc935e6

sreejasahithi requested review from ashishkumar50, priyeshkaratha and sadanand48 January 16, 2026 10:36

priyeshkaratha reviewed Jan 19, 2026

View reviewed changes

aryangupta1998 reviewed Jan 21, 2026

View reviewed changes

Sreeja Chintalapati added 2 commits January 23, 2026 12:19

Added integration test and updated leader find logic

5fbe051

Made minor updates

8dd43f6

sreejasahithi requested review from aryangupta1998 and octachoron January 23, 2026 07:31

Removed noisy warnings

836c36c

aryangupta1998 approved these changes Jan 28, 2026

View reviewed changes

Sreeja Chintalapati added 2 commits February 11, 2026 10:12

Added ScmNodeTarget class to enable querying specific SCM nodes in HA

d9886b2

Removed unused method

2200e34

sumitagrawl reviewed Feb 18, 2026

View reviewed changes

Comment thread ...op-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeCheckSubcommand.java

Sreeja Chintalapati added 3 commits February 24, 2026 18:50

Merge branch 'master' of github.com:apache/ozone into HDDS-14108

1af85e8

simplified SafeModeCheckSubcommand

3e3f9d3

Fixed test failure

742db81

sreejasahithi requested a review from sumitagrawl February 25, 2026 08:12

sumitagrawl approved these changes Feb 25, 2026

View reviewed changes

sumitagrawl merged commit 4779f60 into apache:master Feb 25, 2026
46 checks passed

This was referenced Sep 16, 2021

[Snyk] Security upgrade react-scripts from 3.4.4 to 4.0.0 jojochuang/ozone#16

Closed

[Snyk] Security upgrade react-scripts from 3.4.4 to 4.0.0 jojochuang/ozone#14

Closed

szetszwo reviewed Aug 15, 2026

View reviewed changes

Uh oh!

Conversation

sreejasahithi commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

dombizita commented Jan 10, 2026

Uh oh!

octachoron commented Jan 11, 2026

Uh oh!

octachoron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

octachoron left a comment

Choose a reason for hiding this comment

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

ashishkumar50 Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

ashishkumar50 Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sadanand48 Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

priyeshkaratha left a comment

Choose a reason for hiding this comment

Uh oh!

aryangupta1998 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aryangupta1998 left a comment

Choose a reason for hiding this comment

Uh oh!

sreejasahithi commented Feb 11, 2026

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

szetszwo Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

sreejasahithi commented Jan 9, 2026 •

edited

Loading

sreejasahithi Jan 16, 2026 •

edited

Loading

sadanand48 Jan 14, 2026 •

edited

Loading

sreejasahithi Aug 17, 2026 •

edited

Loading