[shiftstack] Use GitHub for shiftstack-qa clone and fix false-positive test results - #3922
Conversation
|
Skipping CI for Draft Pull Request. |
22a3c0d to
4dd8af5
Compare
imatza-rh
changed the title
b2ab112 to
487590c
Compare
|
This PR is stale because it has been for over 15 days with no activity. |
487590c to
2a0c27d
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 27m 41s |
|
recheck |
2a0c27d to
c01e57b
Compare
c01e57b to
b3ecf49
Compare
b3ecf49 to
609d972
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 10m 58s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 41m 46s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 39m 31s |
f4d8917 to
6397c6c
Compare
|
This PR is stale because it has been for over 15 days with no activity. |
6a7e65d to
8e774f7
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 49m 16s |
|
recheck |
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 38s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 09s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 15m 17s |
|
recheck |
8e774f7 to
aa557b7
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
aa557b7 to
f0abbe1
Compare
…e test results shiftstack-qa moved code review to GitHub PRs (shiftstack/ shiftstack-qa#2). The role default still clones from GerritHub, which is no longer maintained. - Change cifmw_shiftstack_qa_repo default to GitHub - Rename cifmw_shiftstack_qa_gerrithub_change to cifmw_shiftstack_qa_change_ref Also fix a false-positive bug in exec_command_in_pod.yml where oc rsh could return rc=0 before the inner command finished, causing the test to report PASS when the playbook actually failed. Observed in tp!2297 build d078ccbd - the shiftstack test ran for only 145s (vs typical 1.5-2h), OCP was never installed, yet the job reported SUCCESS. The fix runs the inner command detached via nohup and polls a marker file for the exit code. This survives connection drops. Uses kubernetes.core.k8s_exec and k8s_cp modules for setup tasks; the poll step uses oc exec for resilience to transient network errors. Process-crash detection via kill -0 inside the pod catches OOM without polling for 8h. Diagnostic output fetched in rescue/always blocks restores stderr visibility lost by the async approach. Configurable via cifmw_shiftstack_exec_retries (default 5760) and cifmw_shiftstack_exec_delay (default 5s) for an 8h timeout. Increase cifmw_libvirt_manager_compute_memory from 50 to 60 GB in the osasinfra DT scenario. With 24 GB master flavors and ram_allocation_ratio=1.0, two master-sized VMs cannot share a 46 GB compute (2x24576=49152 > 45957 schedulable). 60 GB gives ~56 GB usable (6 GB margin). serval70 has 176 GB free RAM on the host - the increase adds only 28 GB across 3 computes. Related-Issue: #OSPRH-30385 Assisted-By: Claude Code Signed-off-by: Itay Matza <imatza@redhat.com>
f0abbe1 to
c145d21
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 35m 11s |
2 participants
Summary
Three fixes for the shiftstack CI role:
1. Clone shiftstack-qa from GitHub instead of GerritHub
shiftstack-qa moved code review to GitHub PRs (shiftstack/shiftstack-qa#2). The role default still clones from GerritHub, which is no longer maintained. GerritHub main is 3+ commits behind GitHub main - missing the
4-stableOCP channel switch,lb_testsdisable, and.gitreviewremoval.cifmw_shiftstack_qa_repodefault tohttps://github.com/shiftstack/shiftstack-qacifmw_shiftstack_qa_gerrithub_changetocifmw_shiftstack_qa_change_ref- supports both GitHub PR refs and GerritHub change refs2. Fix false-positive and connection-drop bugs in exec_command_in_pod.yml
oc exec/rsh connections to the shiftstackclient pod drop after ~38 minutes during long-running tests. The previous fix (marker file + polling) didn't solve it because the marker file write was chained to the same bash process that dies when the connection drops.
Root cause verified from tp!2297 build d349fb73 inner test log: the test ran for 1h 37m inside the pod (Manila StorageClass failure at 14:20 UTC), but oc exec returned after only 38m 50s. The marker file was never created - 720 retries over 2h all got No such file or directory.
Fix: writes the command to a script file, copies it into the pod, then runs it detached via nohup. The process survives oc exec connection drops. Also kills orphaned background processes from previous invocations to prevent marker file poisoning.
Default polling timeout set to 8h (5760 retries x 5s) to cover the full test suite duration. Configurable via
cifmw_shiftstack_exec_retriesandcifmw_shiftstack_exec_delay, documented in README.md.Verified on the live shiftstackclient pod (serval70) - 6-case Ansible test suite covering success, exit codes 1/42, dollar-sign commands, orphan kill, retry timeout, and nohup survival after oc exec termination.
3. Increase compute VM memory for osasinfra DT scenario
Increase
cifmw_libvirt_manager_compute_memoryfrom 50 to 60 GB indt-osasinfra.yml. With 24 GB master flavors andram_allocation_ratio=1.0, two master-sized OCP VMs cannot share a 46 GB compute node (2x24576=49152 MB > 45957 MB schedulable). 60 GB gives ~51 GB usable, (6 GB margin). serval70 has 176 GB free host RAM - the increase adds only 28 GB across 3 computes.Test plan