Skip to content

Avoid device memory spans in DMA address limit check - #1301

Open
tracycam wants to merge 2 commits into
NVIDIA:mainfrom
tracycam:fix/dma32-zone-device-span
Open

Avoid device memory spans in DMA address limit check#1301
tracycam wants to merge 2 commits into
NVIDIA:mainfrom
tracycam:fix/dma32-zone-device-span

Conversation

@tracycam

Copy link
Copy Markdown

Problem

nv_get_max_sysmem_address() currently uses the largest node_end_pfn() among online NUMA nodes. UVM HMM device-private ranges can stretch a node span to the physical-address ceiling even though those PFNs are not system pages available to GFP_KERNEL.

On a reproduced 52-bit physical-address host:

  • actual System RAM ended at 0x3007fffffff (about 3 TiB)
  • four UVM HMM ranges stretched one node to 0xfffffffffffff
  • the GPUs had a 47-bit DMA mask (0x7fffffffffff)

The node-span comparison therefore selected GFP_DMA32 for ordinary NVIDIA system-memory allocations even though all allocatable RAM was reachable by the devices. Independent CUDA contexts then exhausted the small usable DMA32 pool; the fourth context failed while allocating a 96 MiB system-memory object.

Fix

Compute the maximum address only from managed zones that NV_GFP_KERNEL can allocate from:

  • scan every online NUMA node to preserve unrestricted NUMA fallback
  • scan zone indices through ZONE_NORMAL to cover DMA, DMA32, and Normal
  • skip zones with no buddy-managed pages
  • exclude HighMem, Movable, and Device zones, which plain GFP_KERNEL cannot return

Real managed Normal memory above a device DMA mask still raises the limit and retains the existing DMA32 fallback. The explicit force_dma32_alloc path is unchanged.

This is the driver-side analogue of Linux commit 7170130e4c72, which prevents device-private ranges from inflating the generic max_pfn DMA-reachability check.

Validation

  • make modules -j120 passed for current main (610.57.04) against Linux 7.0.0-29
  • the same kernel-interface change backported to 610.43.03 also built successfully
  • four independent CUDA contexts initialized and ran simultaneously on four 47-bit GA100-class devices
  • a fifth traced 96 MiB allocation reported:
    • gfp=0x6dc0 (GFP_DMA32 bit clear)
    • first_pfn=2138991 (above 4 GiB)
    • nv_alloc_system_pages_ret=0
  • DMA32 pagetype counts were identical before and after the four contexts
  • all four devices ran at 100% utilization during the check
  • no Xid, NV_ERR, kernel warning, or leaked test process remained after cleanup

The helper remains a lockless topology snapshot, matching the existing behavior for concurrent memory hotplug.

@CLAassistant

CLAassistant commented Aug 16, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants