Skip to content

fix(docker): keep libgomp1 in the Ubuntu runtime image (cuDSS threading layer, #1670) - #1888

Open
vgong-nv wants to merge 1 commit into
NVIDIA:mainfrom
vgong-nv:fix/docker-libgomp1
Open

fix(docker): keep libgomp1 in the Ubuntu runtime image (cuDSS threading layer, #1670)#1888
vgong-nv wants to merge 1 commit into
NVIDIA:mainfrom
vgong-nv:fix/docker-libgomp1

Conversation

@vgong-nv

Copy link
Copy Markdown

Closes #1670.

What happens

On the 26.08 -cu12 / -cu13 Ubuntu images, any LP/MIP whose root relaxation reaches the barrier method dies inside cuDSS:

Solving LP root relaxation in concurrent mode
...
FAILED: CUDSS call ended unsuccessfully with status = 3, details: "cudssSetThreadingLayer"
INFO cuopt received signal 17
INFO {'cuopt_complete': -1}
INFO Starting new process with pid 155

The REST client only sees {'error': '<id> was aborted', 'error_result': True}. Small problems pass because dual simplex / PDLP finish the root LP before barrier initialises cuDSS; large ones fail every time. Same model, same machine, 26.04 image: fine.

Why

cuDSS's threading layer (libcudss_mtlayer_gomp.so) dlopens libgomp.so.1, and the 26.08 image does not have it:

$ ldd .../nvidia/cu13/lib/libcudss_mtlayer_gomp.so.0 | grep gomp
libgomp.so.1 => not found
$ dpkg -l libgomp1   # not installed

ci/docker/Dockerfile installs gcc to build psutil and purges it in the same layer (apt-get purge -y --autoremove gcc, since #1602). libgomp1 was only ever present as a dependency of gcc, so the autoremove takes it too. 26.04 had it by that side effect.

Fix

Install libgomp1 explicitly in the same apt-get install as gcc. apt then marks it as manually installed and the purge keeps it. One word in the Dockerfile plus a comment.

Verified the apt semantics on ubuntu:24.04 (the LINUX_VER CI builds with):

install after apt-get purge -y --autoremove gcc
gcc (today) libgomp1 gone
gcc libgomp1 (this PR) ii libgomp1:arm64 14.2.0-4ubuntu2~24.04

Dockerfile.ubi is unchanged: UBI 10's base image already ships libgomp and dnf remove gcc leaves it in place (checked the same way).

Testing

  • The equivalent runtime fix (official 26.8.0-cu12 image + apt-get install libgomp1) has been running on a DGX Spark (GB10, arm64, driver 580) for a week of MIP solves — 67k-binary facility-location models, 45–300 s, REST and gRPC — with no cuDSS failure; the same models abort within a second on the unmodified image.
  • I have not run the full build_images workflow; the change is confined to the apt line.

Not in this PR

A failed cudssSetThreadingLayer currently ends the solver process. Falling back to single-threaded cuDSS would turn a missing library into a slower solve instead of an aborted job; that is a libcuopt change and out of scope here.

cuDSS's threading layer (libcudss_mtlayer_gomp) dlopens libgomp.so.1. The
26.08 Ubuntu images do not have it: ci/docker/Dockerfile installs gcc to build
psutil and purges it with --autoremove in the same layer, and libgomp1 was only
ever present as a dependency of gcc. Any LP/MIP whose root relaxation reaches
barrier then dies in cudssSetThreadingLayer (status 3) and the REST client sees
"<id> was aborted"; small problems pass because simplex/PDLP finish the root LP
first, which is why NVIDIA#1670 looked random.

Install libgomp1 explicitly alongside gcc so apt marks it manual and the purge
keeps it. Verified on ubuntu:24.04: with `gcc` alone the purge removes
libgomp1, with `gcc libgomp1` it stays. Dockerfile.ubi is unchanged — UBI 10's
base already ships libgomp and dnf remove gcc leaves it.

Closes NVIDIA#1670

Signed-off-by: Vincent Gong <vgong@nvidia.com>
@vgong-nv
vgong-nv requested a review from a team as a code owner September 11, 2026 03:38
@vgong-nv
vgong-nv requested a review from bdice September 11, 2026 03:38
@copy-pr-bot

copy-pr-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 32609534-b18e-43f0-9d2d-9f4595354bfa

📥 Commits

Reviewing files that changed from the base of the PR and between 2d964a2 and 6e85224.

📒 Files selected for processing (1)
  • ci/docker/Dockerfile

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The Docker install stage now installs libgomp1 with gcc. Cleanup still removes gcc while retaining libgomp1 for cuDSS threading.

Changes

cuDSS runtime dependency

Layer / File(s) Summary
Install and retain the OpenMP runtime library
ci/docker/Dockerfile
The image build installs libgomp1 with gcc. Cleanup removes gcc and retains libgomp1 for cuDSS execution.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 6e852

The runtime image retains the library required for cuDSS threading after compiler cleanup, with no actionable merge risk identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Linked Issues check ❓ Inconclusive The change in ci/docker/Dockerfile explicitly installs libgomp1 with gcc before gcc is purged. This addresses the reported missing libgomp.so.1 cause for cudssSetThreadingLayer status 3 in… Provide reviewable results for the two specified Docker environments, including the repeated 100-execution test and its segmentation-fault count. The current evidence is insufficient to decide whether the issue's validation requirements are…
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the Docker runtime fix and the specific cuDSS threading-layer issue.
Description check ✅ Passed The description directly explains the cuDSS failure, its libgomp1 cause, the Dockerfile fix, and the validation performed.
Out of Scope Changes check ✅ Passed The pull request changes only ci/docker/Dockerfile. The added package installation and explanatory comments directly support the cuDSS runtime dependency and issue #1670. No unrelated source, image,…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Full details: Linked Issues check

Explanation

The change in ci/docker/Dockerfile explicitly installs libgomp1 with gcc before gcc is purged. This addresses the reported missing libgomp.so.1 cause for cudssSetThreadingLayer status 3 in issue #1670. The available evidence does not establish validation in nvidia/cuopt:26.8.0a-cuda13.3-py3.14 and 26.6.0-cuda13.2-py3.14 on RTX PRO 6000 Blackwell systems, or zero segmentation faults across 100 repeated executions. The summary also states that the full build_images workflow was not run.

Resolution

Provide reviewable results for the two specified Docker environments, including the repeated 100-execution test and its segmentation-fault count. The current evidence is insufficient to decide whether the issue's validation requirements are complete.

✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@ramakrishnap-nv ramakrishnap-nv added bug Something isn't working non-breaking Introduces a non-breaking change labels Sep 11, 2026
@ramakrishnap-nv

ramakrishnap-nv commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the writeup and validation, Vincent — added the bug + non-breaking labels for you.

Different mechanism than #1219/#1768/#1774 (wheel-bundled libgomp), same underlying gap: nothing treats "cuDSS needs a real system libgomp" as a first-class dependency. #1883 fixes the #1219 crash mechanism directly (RTLD_NODELETE on cuDSS's threading-layer plugin); this PR closes the corresponding Docker-side gap. Complementary, not overlapping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working non-breaking Introduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] FAILED: CUDSS call ended unsuccessfully with status = 3, details: "cudssSetThreadingLayer"

2 participants