kvm: trim whitespace around backup repository mount options on restore

[stag7824]

Description

Fixes #14013

A backup repository whose mount options carry stray whitespace, for example vers=4.1 ,
backs up fine but fails to restore:

mount.nfs: an incorrect mount option was specified for
/usr/share/cloudstack-agent/tmp/csbackup...

LibvirtRestoreBackupCommandWrapper.mountBackupDirectory builds the mount command as an argv
list and runs it without a shell:

if (StringUtils.isNotBlank(mountOptions)) {
    mountCmd.add("-o");
    mountCmd.add(mountOptions);
}
Script.executeCommand(mountCmd.toArray(new String[0]));

Script.executeCommand(String...) hands each element to ProcessBuilder verbatim, so mount
receives the literal option list vers=4.1 . libmount splits that list on commas only, so the
blank stays glued to the last option and the kernel rejects it as part of the option value.

Backup, delete and stats do not hit this today because they pass the options to nasbackup.sh,
which expands them unquoted:

mount -t ${NAS_TYPE} ${NAS_ADDRESS} ${mount_point} $([[ ! -z "${MOUNT_OPTS}" ]] && echo -o ${MOUNT_OPTS})

Shell word splitting drops the blank there, which is why restore is the only operation that
fails: it is the only one of the four that builds the mount command in Java.

Fixed by normalising the option list before it is used. Each comma-separated option is trimmed
and empty ones are dropped. Only the whitespace around the delimiters is removed; the option
text itself is passed through unchanged, since deciding whether an option is well formed
belongs to the API layer rather than the agent.

The normalisation happens before the cifs branch appends ,nobrl. Trimming afterwards
would leave vers=3.0 ,nobrl, moving the blank into the middle of the list where libmount
cannot ignore it either.

Relationship to #14009

#14009 stops this class of value from being stored: it rejects whitespace per option in
AddBackupRepositoryCmd / UpdateBackupRepositoryCmd, and it quotes "${MOUNT_OPTS}" in
nasbackup.sh. That is the right place for validation and I have deliberately not duplicated
any of it here — the two changes touch no common files.

They are complementary rather than alternatives, because #14009 validates on the way in and
cannot clean rows that are already in the database. On an existing deployment such as the one
in #14013, the stored vers=4.1 survives, and once MOUNT_OPTS is quoted it will break the
script-based operations too, not just restore. Normalising at the point of use fixes those
repositories without a schema migration.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• Minor

How Has This Been Tested?

Added seven unit tests to LibvirtRestoreBackupCommandWrapperTest that invoke
mountBackupDirectory and assert on the exact -o argument handed to mount:

mount options	repo type	-o argument	fails without the fix
vers=4.1	nfs	vers=4.1	yes
vers=4.1 , soft	nfs	vers=4.1,soft	yes
vers=4.1,,soft	nfs	vers=4.1,soft	yes
vers=3.0	cifs	vers=3.0,nobrl	yes
username=some user	cifs	username=some user,nobrl	no
	nfs	no -o at all	no
	cifs	nobrl	no

The last three already hold today. They are there to pin behaviour the fix must not change:
that trimming is scoped to the delimiters rather than stripping whitespace throughout, and
that an all-whitespace value still collapses to no options at all (or to bare nobrl for
cifs) instead of becoming an empty option.

mvn test -pl plugins/hypervisors/kvm -Dtest=LibvirtRestoreBackupCommandWrapperTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0

Removing just the normalizeMountOptions call and rerunning gives 4 failures, each showing
the blank that mount rejects:

expected:<vers=4.1[]>        but was:<vers=4.1[ ]>
expected:<[vers=4.1,soft]>   but was:<[ vers=4.1 , soft ]>
expected:<vers=4.1,[]soft>   but was:<vers=4.1,[,]soft>
expected:<vers=3.0[],nobrl>  but was:<vers=3.0[ ],nobrl>

The last of those is the ordering point: the blank lands in the middle of the list once
nobrl is appended after it.

I do not have a KVM zone with a NAS repository available, so this is unit-level only; the
mount argument that the kernel rejects is asserted directly rather than end to end.

The same construction is present on 4.20. I targeted 4.22 because that is the version the
issue is reported against, but happy to retarget if it should land on 4.20 first.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

• In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
• Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
• Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
• Be sure to read the CloudStack Coding Conventions.
  Apache CloudStack is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
  Slack: https://apachecloudstack.slack.com/

[weizhouapache]

@stag7824
have you verified it on a cloudstack environment ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 17.86%. Comparing base (a9b2f33) to head (afdeb67).

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #14096      +/-   ##
============================================
- Coverage     17.86%   17.86%   -0.01%     
- Complexity    16037    16040       +3     
============================================
  Files          5928     5928              
  Lines        534479   534490      +11     
  Branches      65410    65412       +2     
============================================
- Hits          95468    95460       -8     
- Misses       428173   428199      +26     
+ Partials      10838    10831       -7     

Flag	Coverage Δ
uitests	4.02% <ø> (ø)
unittests	18.92% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[stag7824]

Not on a full CloudStack zone, no — I don't have a KVM host with a NAS backup repository, and I didn't want to imply otherwise in the description.

What I could do is exercise the real code path against a real CIFS server, with nothing mocked. I ran mountBackupDirectory itself by reflection — real Script, real ProcessBuilder, real mount.cifs — against a samba share holding a file that exists only on the share, and checked whether that file could be read back afterwards:

mount options	with this PR	without it
guest,vers=3.0 (trailing space)	mounted, payload read back	not mounted
guest,vers=3.0 (no space, control)	mounted, payload read back	mounted, payload read back

The control row is there to show the harness isn't just reporting success, and that the change is inert for well-formed options.

The command that the unfixed path actually hands to mount, captured from the process table:

/sbin/mount.cifs //127.0.0.1/backup /tmp/csbackup.epddp75190 -o rw,vers=3.0 ,nobrl

which is the case in the description — nobrl is appended after the blank, so it ends up mid-list rather than at the end.

Worth flagging separately, because it changes how this looks in the field: Script.executeCommand(String...) returns null on failure rather than throwing, and mountBackupDirectory ignores the return value. So a failed mount isn't detected — restore carries on against an empty temp directory and fails later with a confusing error rather than a mount error. That's pre-existing and I've deliberately left it out of this PR, but happy to raise it as its own issue if you'd like.

What I still can't cover locally is the full path: management server, agent, a real backup and restore of a VM volume. If that's needed before this can go in, could someone with a NAS repository confirm it? Happy to adjust the approach — including moving the normalisation elsewhere — if you'd rather it not live in the agent.