Report work_mem wanted by a sort that spilled to disk

[Alena0704]

"Memory wanted" in EXPLAIN ANALYZE is built from instrument->workmemwanted, the work_mem an operator needed to avoid spilling. Hash aggregation and hash join set it, but the sort no longer does: a sort that spilled gigabytes to disk contributed nothing, and EXPLAIN advised less memory than the query had already been given.
This is a backport of the Greenplum 7 fix, greenplum-db/gpdb@fe0a9ddbdd (#13590). Cloudberry forked before it landed, so the call that hands the sort its Instrumentation is still disabled in nodeSort.c under a GPDB_12_MERGE_FIXME; the backport enables it again and is adapted to PostgreSQL 14. A second commit extends it to parallel sort workers: each worker hands its estimate over through the statistics it already keeps in shared memory, and the leader reports the largest one.

Reproduction :

CREATE TABLE memwanted_sort (id int, pad text) DISTRIBUTED BY (id);
INSERT INTO memwanted_sort SELECT g, repeat(chr(97 + g % 26), 200)
  FROM generate_series(1, 100000) g;
ANALYZE memwanted_sort;

SET statement_mem = '2MB';
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
  SELECT * FROM memwanted_sort ORDER BY pad, id;

Before : the sort writes to disk and says nothing about it — no Memory wanted line at all, and the slice only reports how much work_mem it got.

 Gather Motion 3:1  (slice1; segments: 3) (actual rows=100000 loops=1)
   Merge Key: pad, id
   ->  Sort (actual rows=33462 loops=1)
         Sort Key: pad, id
         Sort Method:  external merge  Disk: 21696kB
         ->  Seq Scan on memwanted_sort (actual rows=33462 loops=1)
   (slice1)    Executor memory: 1707K bytes avg x 3x(0) workers, 1707K bytes max (seg0).  Work_mem: 1707K bytes max.
 Memory used:  2048kB
 Execution Time: 193.199 ms

After: the sort reports what it would have needed.

 Gather Motion 3:1  (slice1; segments: 3) (actual rows=100000 loops=1)
   Merge Key: pad, id
   ->  Sort (actual rows=33462 loops=1)
         Sort Key: pad, id
         Sort Method:  external merge  Disk: 21696kB
         ->  Seq Scan on memwanted_sort (actual rows=33462 loops=1)
 * (slice1)    Executor memory: 1707K bytes avg x 3x(0) workers, 1707K bytes max (seg0).  Work_mem: 1707K bytes max, 10686K bytes wanted.
 Memory used:  2048kB
 Memory wanted:  10885kB
 Execution Time: 186.126 ms

With statement_mem = '10MB' the sort no longer spills, and there is nothing left to ask for.

SET statement_mem = '10MB';
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
  SELECT * FROM memwanted_sort ORDER BY pad, id;

 Gather Motion 3:1  (slice1; segments: 3) (actual rows=100000 loops=1)
   Merge Key: pad, id
   ->  Sort (actual rows=33462 loops=1)
         Sort Key: pad, id
         Sort Method:  quicksort  Memory: 29747kB
         ->  Seq Scan on memwanted_sort (actual rows=33462 loops=1)
  * (slice1)    Executor memory: 8334K bytes avg x 3x(0) workers, 8366K bytes max (seg0).  Work_mem: 8366K bytes max.
 Memory used:  10240kB
 Execution Time: 161.383 ms

While testing the sort fix I found that Incremental Sort mishandles memory in two related ways, so this PR carries two more commits:

• It ignored statement_mem. The node always sized its sorts from the work_mem GUC, and it was counted among the cheap operators that get 100kB, so it was also left out when the memory need of the statement was worked out.
• It reported nothing. The node sorts the rows group by group, each group in a sort of its own, and none of those figures reached EXPLAIN ANALYZE.

Reproduction :

create table inc5 (id bigint, grp int, pad text) distributed by (id);
create table inc10(id bigint, grp int, pad text) distributed by (id);
insert into inc5  select g, g % 5,  repeat(chr(97 + g % 26), 100) from generate_series(1, 2000000) g;
insert into inc10 select g, g % 10, repeat(chr(97 + g % 26), 100) from generate_series(1, 4000000) g;
create index on inc5 (grp);
create index on inc10 (grp);
analyze inc5;
analyze inc10;

set optimizer = off;               -- only the Postgres planner builds an Incremental Sort
set enable_incremental_sort = on;  -- off by default in Cloudberry
set enable_sort = off;
set enable_seqscan = off;
set enable_bitmapscan = off;
set statement_mem = '2MB';

explain (analyze, costs off, timing off) select * from inc5  order by grp, pad;
explain (analyze, costs off, timing off) select * from inc10 order by grp, pad;

Before

 ->  Incremental Sort (actual rows=667670 loops=1)
       Pre-sorted Groups: 5  Sort Method: external merge  Average Disk: 16102kB  Peak Disk: 16128kB
   (slice1)    Executor memory: 35545K bytes avg x 3x(0) workers, 35545K bytes max (seg0).
 Memory used:  2048kB

Raising statement_mem to 64MB changes nothing. Each group is measured on its own, so the advice does not grow with the number of groups.:

 ->  Incremental Sort (actual rows=667670 loops=1)
       Pre-sorted Groups: 5  Sort Method: external merge  Average Disk: 16102kB  Peak Disk: 16128kB
 Memory used:  65536kB

After

inc5, statement_mem = '2MB' — the node stays within its share and says what it needs:

 ->  Incremental Sort (actual rows=667670 loops=1)
       Pre-sorted Groups: 5  Sort Method: external merge  Average Disk: 16166kB  Peak Disk: 16192kB
 * (slice1)  Executor memory: 2045K bytes ...  Work_mem: 1622K bytes max, 42775K bytes wanted.
 Memory used:  2048kB
 Memory wanted:  42974kB

inc10 — twice the data and twice the groups, but groups of the same size, so the advice is the same:

 ->  Incremental Sort (actual rows=1334931 loops=1)
       Pre-sorted Groups: 10  Sort Method: external merge  Average Disk: 16156kB  Peak Disk: 16192kB
 * (slice1)  Executor memory: 1939K bytes ...  Work_mem: 1622K bytes max, 42759K bytes wanted.
 Memory wanted:  42958kB

Give inc5 the 42MB it asked for and nothing is written to disk.

 ->  Incremental Sort (actual rows=667670 loops=1)
       Pre-sorted Groups: 5  Sort Method: quicksort  Average Memory: 41738kB  Peak Memory: 41789kB
 Memory used:  65536kB

The busiest group peaks at 41789kB against the 42775kB predicted — within 3%.

Type of Change

• Bug fix (non-breaking change)
• New feature (non-breaking change)
• Breaking change (fix or feature with breaking changes)
• Documentation update

Breaking Changes

Test Plan

• Unit tests added/updated
• Integration tests added/updated
• Passed make installcheck
• Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

• Followed contribution guide
• Added/updated documentation
• Reviewed code for security implications
• This PR contains AI-assisted code generation
• Requested review from cloudberry committers

Additional Context

CI Skip Instructions

---