Skip to content

fix: prevent OOM in Automodel recipes - #4034

Merged
sharonyu-115 merged 9 commits into
NVIDIA-NeMo:mainfrom
jQizhang:fix/automodel-oom-recipes
Sep 8, 2026
Merged

fix: prevent OOM in Automodel recipes#4034
sharonyu-115 merged 9 commits into
NVIDIA-NeMo:mainfrom
jQizhang:fix/automodel-oom-recipes

Conversation

@jQizhang

@jQizhang jQizhang commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Updates the affected Automodel recipes and test suites with memory-safe configurations.

Fixes #3842

@jQizhang
fix: prevent OOM in Nemotron AutoModel recipes
821e875 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang
fix: prevent OOM in Nemotron Super AutoModel recipe
55806b8 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang
fix: restore Nemotron recipe runtime settings
e41d6fd 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang
fix: align Nemotron GRPO recipe topology naming
8850f5a 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang
fix: version Nemotron Super AutoModel recipe
859fa78 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang
Merge origin/main into fix/automodel-oom-recipes
9945af8 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang
jQizhang requested review from a team as code owners September 7, 2026 02:38
@copy-pr-bot

copy-pr-bot Bot commented Sep 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@jQizhang
ci: update secret baseline for renamed recipe
7280c70 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang

jQizhang commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7280c70

@jQizhang
fix: update references to versioned recipes
f208ac8 
Signed-off-by: larkzhang-nv <larkz@nvidia.com>
@jQizhang jQizhang added the CI:L2 Run doctests, unit tests, functional tests, and convergence tests label Sep 7, 2026
@jQizhang
jQizhang requested a review from a team as a code owner September 7, 2026 03:34
@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Sep 7, 2026
@jQizhang

jQizhang commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f208ac8

@jQizhang jQizhang added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L2 Run doctests, unit tests, functional tests, and convergence tests labels Sep 7, 2026
jQizhang
jQizhang commented Sep 7, 2026
View reviewed changes

@jQizhang jQizhang left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #4034 review — fix: prevent OOM in Automodel recipes

Reviewed by a 5-agent team (rl-expert, test-agent, bug-finder, design-reviewer, devil-advocate). No merge conflicts. Confirmed non-trivial: genuine memory-tuning across 4 recipe YAMLs plus a coordinated .vN rename pass and the common-tq.env fix needed to keep TQ coverage matching post-rename.

No design-level surface (diff is entirely YAML/docs/bash — no .py files touched) — design-reviewer's LGTM confirmed by devil-advocate.

Linter (pre-commit run --all-files, including minimize-check llm recipes): all hooks passed.

1 finding needs author input before merge (test-rigor regression with no stated justification), 2 low-severity suggestions below.

Generated by Claude Code

Comment thread tests/test_suites/llm/dpo-nanov3-30B3AB-1n8g-fsdp8ep8-automodel.v2.sh
Comment thread tests/test_suites/llm/common-tq.env
Comment thread tests/test_suites/llm/common-tq.env
sharonyu-115
sharonyu-115 requested changes Sep 7, 2026
View reviewed changes
Comment thread tests/test_suites/llm/dpo-nanov3-30B3AB-1n8g-fsdp8ep8-automodel.v2.sh
Comment thread examples/configs/recipes/llm/grpo-nanov3-30BA3B-1n8g-fsdp2.v2.yaml
sharonyu-115
sharonyu-115 approved these changes Sep 8, 2026
View reviewed changes

@sharonyu-115 sharonyu-115 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@sharonyu-115
sharonyu-115 enabled auto-merge (squash) September 8, 2026 03:04
@jQizhang

jQizhang commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 01c2e08

@sharonyu-115
sharonyu-115 merged commit 13182a5 into NVIDIA-NeMo:main Sep 8, 2026
84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sharonyu-115 sharonyu-115 sharonyu-115 approved these changes

Assignees

No one assigned

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Track nightly OOMs after the Automodel r0.6.0 upgrade

2 participants

@jQizhang @sharonyu-115