test: [3/3] add L3 Nano 4B Gym training E2E

[yaoyu-33]

Purpose

Add the final L3 layer of the Gym/RL acceptance-test series: a real one-step NeMo RL training job driven by a NeMo Gym environment and verifier.

This covers the gap between L2 rollout compatibility and successful optimizer/refit execution with a bounded deterministic workload.

Coverage

• loads nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 at exact revision d552708a9d575fa8d4a690b988fd870d65279f98 for FSDP2 policy, tokenizer, and vLLM
• runs on a two-H100 functional runner with colocated TP1 vLLM and two-rank FSDP2
• starts Gym math_with_judge in deterministic no-judge mode
• loads a checked-in two-row NemoGymDataset fixture with one accepted and one rejected rollout
• requires exact train reward min/max/mean of 0/1/0.5 and signed Reinforce++ advantages
• completes exactly one optimizer step with finite loss, finite nonzero gradient norm, and finite positive learning rate
• requires nonempty explicit generation-worker refit acknowledgements through refit/generation_workers_updated
• runs post-refit validation and requires exact accuracy 0.5
• checks finite bounded token-probability parity and positive generation/validation timing

The test exposed generation/refit correctness gaps. This PR normalizes temperature=0 to vLLM's unscaled/unfiltered logprob semantics across AutoModel, Megatron, and DTensor paths; scopes tiny-positive clamping to vLLM/Dynamo; rejects unsupported non-unit temperature on fused Megatron logprobs; allows generation configs that omit temperature; and makes direct vLLM/TRT refit success require nonempty, explicit worker acknowledgements.

Validation

Current head: 4e6e44ecde057de137ae9cafc9e24d0550a5dce7

• Exact-head x2-H100 Gym training E2E passed: https://github.com/NVIDIA-NeMo/RL/actions/runs/34014189087
  • train/grad_norm: finite and positive (110.023094...)
  • train/lr: finite and positive (4.999999987e-07)
  • train reward min/max/mean: 0 / 1 / 0.5
  • refit/generation_workers_updated: 2.0
  • post-refit validation/accuracy: 0.5
  • validation time: 3.264s
• All three current-head Vllm unit-test shards passed: https://github.com/NVIDIA-NeMo/RL/actions/runs/34016095608
• Complete exact-head L0/H100 run: 22 of 23 unit-test shards passed and all coverage uploads succeeded: https://github.com/NVIDIA-NeMo/RL/actions/runs/34049400947
• Codecov patch coverage passed at 94.12% (32 hit / 2 miss; target 80%)
• Ruff/lint, script coverage, YAML/JSONL parsing, shell syntax, Python compile, git diff --check, and DCO sign-off: passed
• Multiple independent review/fix rounds; final evidence review found no P0-P2 issues

The functional job took about 12 minutes; the training process itself took about 4.5 minutes. The workflow's fresh exact-head image build took additional time.

Explicit scope boundaries

• This proves successful rollout, verifier, optimizer, refit-acknowledgement, and post-refit validation execution; it does not checksum model parameters before/after optimizer or refit.
• This configuration exercises colocated legacy IPC refit, not the non-colocated reload_weights transport.
• It covers one environment/verifier and one deterministic two-row workload; broad environment coverage belongs to L1/L2, and this is not a convergence or performance-regression test.
• GB200 has not been validated by this PR.
• The only failed shard in the exact-head L0 run is the unrelated main-branch Megatron model-owned-cp-slicing failure tracked by ci: Bump Megatron-Bridge to 5ed9799  #4022; that fix is intentionally not mixed into this PR.

Three-PR series

1. test: [1/3] add L1 environment verifier acceptance coverage Gym#3127 - L1 environment/verifier acceptance coverage
2. test: [2/3] add L2 Gym-to-RL rollout acceptance #4014 - L2 Gym-to-RL rollout acceptance
3. this PR - L3 full RL training E2E

Draft: the author will review before this is ready for merge.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yaoyu-33]

/ok to test 585004a

[yaoyu-33]

Targeted L3 CI (L1 / Gym_Training / H100 / main image): https://github.com/NVIDIA-NeMo/RL/actions/runs/33950761022

[yaoyu-33]

The first targeted dispatch failed before tests because reusing image_tag=main also sets FAST=1; the runner then looked for the run-id image and never started training. No code failure occurred. Corrected run (branch image build, FAST unset): https://github.com/NVIDIA-NeMo/RL/actions/runs/33951100603

[yaoyu-33]

/ok to test d961928

[yaoyu-33]

Corrected L3 x2-H100 run for : https://github.com/NVIDIA-NeMo/RL/actions/runs/33955622296\n\nThe previous real run exposed two test-contract bugs before training: it requested four Ray GPUs on the two-GPU functional runner, and the inherited train dataset remained OpenMathInstruct-2. The current head uses a 1-train + 1-non-colocated-vLLM split and explicitly binds the two-row NemoGymDataset fixture. Independent final reviews found no remaining P0/P1/P2.

[yaoyu-33]

Correction: the new reviewed head is d961928. Corrected L3 x2-H100 run: https://github.com/NVIDIA-NeMo/RL/actions/runs/33955622296. The current head uses 1 training GPU plus 1 non-colocated vLLM GPU and explicitly binds the two-row NemoGymDataset fixture.

[yaoyu-33]

/ok to test 669c8c7

[yaoyu-33]

Targeted L3 x2-H100 CI for the reviewed two-GPU colocated FSDP2 head: https://github.com/NVIDIA-NeMo/RL/actions/runs/33983791731

[yaoyu-33]

/ok to test d759166

[yaoyu-33]

Targeted L3 x2-H100 CI for the reviewed finite, reproducible training-sampling fix: https://github.com/NVIDIA-NeMo/RL/actions/runs/33989221264

[yaoyu-33]

/ok to test 174f3d3

[yaoyu-33]

/ok to test 9b35225

[yaoyu-33]

Latest-head x2-H100 Gym training E2E passed: https://github.com/NVIDIA-NeMo/RL/actions/runs/33997578151 (target job L1_Functional_Tests_Gym_Training, ~18 minutes). A complete L0 unit matrix is now running at https://github.com/NVIDIA-NeMo/RL/actions/runs/34005722087.