IntrusiveAllocator: heap corruption when large allocations interleave with MemoryBlock slot reuse (crash in Object::unref)

[HouWei-17]

Environment

• VSG 1.1.15 (vcpkg, x64-windows), MSVC 19.44, Windows 10/11
• Vulkan driver: NVIDIA (RTX 3060 Ti, driver 595.79)

Summary

Application crashes with an access violation inside vsg::Object::unref (calling the vtable destructor on a corrupted/freed object) during vsg::Builder::createQuad. The affected object's vtable/refcount bytes are overwritten with values that look like IntrusiveAllocator::MemoryBlock slot metadata (observed pattern: 0x0002 0x0c91 0xffffffff). The corruption is deterministic for a given build but moves/disappears when the heap layout changes (e.g. when a debugger or procdump is attached), which points to free-list corruption.

Repro pattern (full application; minimal repro not yet isolated)

1. Create a vsgXchange::all reader tree inside a per-call helper function and destroy it each time an image is loaded (allocates ~hundreds of objects, then frees them).
2. Allocate a large vsg::Data (e.g. a 4 MB ubvec4Array2D) and feed it to vsg::Builder::createQuad (textured quad).
3. Repeat loads / rebuild the scene graph. Crash occurs in Object::unref of a still-referenced object during a later createQuad call.

Investigation notes

• IntrusiveAllocator::deallocate(ptr, size) locates the owning block by pointer range (memoryBlocks.upper_bound(ptr) then --itr) and runs MemoryBlock::deallocate(ptr, size) before consulting largeAllocations. A large allocation (made via operator new when size > maximumAllocationSize) whose address happens to fall within a MemoryBlock's address range will be treated as a slot in that block, marking an unrelated slot free and corrupting the free list.
• Instrumentation showed a live object's reference count dropping 2 -> 0 during Builder::createQuad with no matching unref in application code.
• Workaround that eliminates the crash: replace Allocator::instance() with a pass-through allocator using plain ::operator new/delete (no MemoryBlocks). With that, no corruption is observed.

Expected

Either the deallocation path should never interpret large-allocation pointers as block slots, or large allocations should be range-checked against the actual MemoryBlock extents before slot math. A defensive check would also help: IntrusiveAllocator::deallocate validating that ptr is inside [block.start, block.start + block.size) before computing a slot index.

Additional context

Reproducing reliably in Release builds; Debug builds mask the corruption (different heap layout / allocator validation).

[robertosfield]

It would be best to create a unit test in vsgExamples for this issue. Could you suggest a really simply code block that would reveal the problem? Would something as simple as:

   {
      auto large_allocation vsg::ubvec4Array2D(x,y);
    } // out of scope gets deleted

Be sufficient? What values of x,y above are you seeing the problem?

[robertosfield]

I have created a VSG branch that adds extras checks in to the IntrutrusiveAllocation::deallocate(void* ptr, std::size_t size) method to see which branches are being executed and whether any large allocations are being mis-handled:

https://github.com/vsg-dev/VulkanSceneGraph/tree/IntrusiveAllocatorLargeAllocationInvestigation

I have tried with some large paged databases I have that do lots of small and large allocations but as yet haven't triggered any obvious problematic usage. I think there are inefficiencies in the code that I'll revisit but so far haven't found a usage case that triggers what looks like an error.

With one very large paged database I do see a mismatch in number large and small allocations and deallocations on exit that suggests there is data that is remaining cached and not cleaned up. I think this is a separate issue though and not related to the one discussed above.

This branch outputs quite a bit to the console when database paging as the memory is continuously being allocated and deallocated, output looks like:

IntrusiveAllocator::MemoryBlock::deallocate((0x7efd1afbca80, 0) OUTWITH block : 0x7efdc8173650
info: B largeAllocation not flagged as being in MemoryBlock
IntrusiveAllocator::MemoryBlock::deallocate((0x7efcfee6fa80, 0) OUTWITH block : 0x6468b41f6320
info: B largeAllocation not flagged as being in MemoryBlock
IntrusiveAllocator::MemoryBlock::deallocate((0x7efd618b1600, 0) OUTWITH block : 0x7efdc8173650
info: B largeAllocation not flagged as being in MemoryBlock
IntrusiveAllocator::MemoryBlock::deallocate((0x7efd1f46a410, 0) OUTWITH block : 0x7efdc8173650
info: B largeAllocation not flagged as being in MemoryBlock
IntrusiveAllocator::~IntrusiveAllocator()
   num_zero_allocations 0
   num_nullptr_deallocations 126
   num_small_allocations 3078113
   num_large_allocations 4539
   num_small_deallocations 2074678
   num_large_deallocations 2386
   num_poor_large_deallocation_checks 2386
   num_no_deallocations 0
   num_erroneous_large_deallocations 0

The last lot of num_* entries is in the destructor.

@80520997 could you test this branch with your data and let me know what the output is?

[HouWei-17]

Thanks for creating the investigation branch and for the detailed notes. I've tested it with our application and can confirm the issue reproduces on your branch — and importantly, every large allocation deallocation hits the problematic path.

Test setup

• Branch: IntrusiveAllocatorLargeAllocationInvestigation @ 0705ef4
• Platform: Windows 10/11, MSVC 19.44, x64, NVIDIA RTX 3060 Ti
• App: our GIS client (vsgXchange reads a 1024x1024 PNG → vsg::ubvec4Array2D ~4 MB, then vsg::Builder::createQuad, scene rebuild, repeated)

Repro loop (via a --stress N mode we added)

1. Load 1024x1024 PNG via a per-call vsgXchange reader tree (created & destroyed in a helper → lots of small alloc/dealloc)
2. Feed the ~4 MB ubvec4Array2D to Builder::createQuad
3. Rebuild scene + viewer->compile()
4. Repeat 200 times

Results (200 iterations)

Stable 1009 occurrences of:

IntrusiveAllocator::MemoryBlock::deallocate((0x..., 0) OUTWITH block : 0x...
info: B largeAllocation not flagged as being in MemoryBlock

Branch split: B=850, D=20, F=139.

Destructor statistics:

num_small_allocations 319714
num_large_allocations 1009
num_small_deallocations 319714
num_large_deallocations 1009
num_poor_large_deallocation_checks 1009
num_erroneous_large_deallocations 0

What this tells us

• num_poor_large_deallocation_checks == num_large_deallocations == 1009: every large allocation's address falls within some MemoryBlock's [memory, memoryEnd) range and gets sent into MemoryBlock::deallocate's OUTWITH path before falling back to the largeAllocations branch.
• On your branch the extra checks correctly reject it (erroneous=0) so it doesn't crash — which is exactly the mechanism we analysed: on v1.1.15 without those checks, the same overlap walks into MemoryBlock::deallocate internals, computes a bogus C = ptr - memory - 1 slot, marks an unrelated live slot free, corrupts the free list, and a still-live object's refcount drops to 0 → Object::unref crash in a later createQuad.
• Our previous workaround (a pass-through allocator using plain new/delete) eliminated the crash, consistent with this.

Minimal repro suggestion

A loop like this should hit the address-overlap condition on your branch:

for (int i = 0; i < 200; ++i) {
    auto io = vsgXchange::all::create();                       // many small allocations
    auto data = vsg::read("1024x1024.png", io);               // ~4 MB large allocation
    auto quad = vsg::Builder::createQuad(...);                // scene allocations
}   // io/data/quad released here

Happy to provide the full 2236-line log if useful — just let me know.

[robertosfield]

I don't agree with your assessment:

num_poor_large_deallocation_checks == num_large_deallocations == 1009: every large allocation's address falls within some MemoryBlock's [memory, memoryEnd) range and gets sent into MemoryBlock::deallocate's OUTWITH path before falling back to the largeAllocations branch.

The OUTWITH code path is being report because MemoryBlock::within(ptr) is correctly detecting that a large allocation ptr is outside the [memory, memoryEnd] range.

A MemoryBlock [memory, memoryEnd] marks the extents of a single memory allocation, as this memory is already allocated then the large allocations won't be within this and visa versa, if there were overlapping it'd suggest that std::new/delete are failing.

It's perfectly possible and legal for a large allocation to fall between adjacent MemoryBlocks, just not within a a MemoryBlock.

---

With your code snippet, does the code ever crash?

[robertosfield]

I have taken your "psuedo" code that doesn't come close to compiling and transformed it into an actual test program that can be run and checked it into IntrusiveAllocatorTest branch of vsgExamples

https://github.com/vsg-dev/vsgExamples/tree/IntrusiveAllocatorTest

The new vsgallocationtests test program can be found in the vsgExamples/tests/vsgalloactionstests directory. can be found in vsgExamples/tests/vsgallocationtests/vsgallocationtests.cpp

When I run it, it completes successfully and finishes with the console output (running against the https://github.com/vsg-dev/VulkanSceneGraph/tree/IntrusiveAllocatorLargeAllocationInvestigation branch of the VSG):

info: data = ref_ptr<vsg::Data>(vsg::ubvec4Array2D 0x71486fa59018), quad = ref_ptr<vsg::Node>(vsg::StateGroup 0x714870a5a438)
IntrusiveAllocator::MemoryBlock::deallocate((0x71486ce28010, 0) OUTWITH block : 0x5866afc7c6d0
info: D largeAllocation not flagged as being in MemoryBlock.
IntrusiveAllocator::MemoryBlock::deallocate((0x5866afca3c10, 0) OUTWITH block : 0x5866afc7c6d0
info: D largeAllocation not flagged as being in MemoryBlock.
IntrusiveAllocator::MemoryBlock::deallocate((0x5866b07bbd30, 0) OUTWITH block : 0x5866afc7c6d0
info: D largeAllocation not flagged as being in MemoryBlock.
IntrusiveAllocator::MemoryBlock::deallocate((0x5866b0a81d90, 0) OUTWITH block : 0x5866afc7c6d0
info: D largeAllocation not flagged as being in MemoryBlock.
info: data = ref_ptr<vsg::Data>(vsg::ubvec4Array2D 0x71486fa59018), quad = ref_ptr<vsg::Node>(vsg::StateGroup 0x714870a5a438)
IntrusiveAllocator::MemoryBlock::deallocate((0x71486ce28010, 0) OUTWITH block : 0x5866afc7c6d0
info: D largeAllocation not flagged as being in MemoryBlock.
Time 13389.5ms iterationCount = 200
IntrusiveAllocator::~IntrusiveAllocator()
   num_zero_allocations 0
   num_nullptr_deallocations 0
   num_small_allocations 32803
   num_large_allocations 800
   num_small_deallocations 32803
   num_large_deallocations 800
   num_poor_large_deallocation_checks 800
   num_no_deallocations 0
   num_erroneous_large_deallocations 0

At this point I can't see any erroneous appearing. The deallocate(..) code could likely be refined to minimize the redundant checks and improve the performance, but I don't yet have anything that is an actual bug that I can investigate.

[robertosfield]

@80520997 I have made a small performance refinement to IntrusiveAllocator::deallocate(void* ptr, std::size_t size) by moving the check for the IntrusiveAllocator::largeAllocations container to the start of the IntrusiveAllocator::deallocate(..) method so it's checked and handled before the memoryBlocks containers are checked.

These changes make are now merged with VSG master: #1734

When running the vsgallocationtests test program I get a small performance improvement and should avoid the case where it looks like memoryBlocks is being used but is actually just a wasteful check rather than a bug.

I don't rule out there might a bug lurking in InstrusiveAllocator, but there isn't any yet related to this thread has given me anything that reproduces an actual bug, but I can't chase imaginary problems, so I'll close this Issue. If you can establish a bug such as by refining the vsgallocationstests program I added to the IntrusiveAllocatorTest branch of vsgExamples then we can reopen this Issue or create a new one with specific guidance on nature of the bug. You'll need to use VSG master for these tests as well as the refinement I just checked in will alter the behavior of IntrusiveAllocator in terms of code paths executed.