Skip to content

[Feature](lambda) Support some map lambda functions - #67284

Draft
morrySnow wants to merge 5 commits into
apache:masterfrom
morrySnow:map-lambda
Draft

[Feature](lambda) Support some map lambda functions#67284
morrySnow wants to merge 5 commits into
apache:masterfrom
morrySnow:map-lambda

Conversation

@morrySnow

@morrySnow morrySnow commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: #67045, #67047

Doc: apache/doris-website#4085

Problem Summary:

This PR adds MAP lambda support under the Nereids planner.

Supported higher-order functions:

  • map_filter((k, v) -> predicate, map)
  • map_exists((k, v) -> predicate, map)
  • map_all((k, v) -> predicate, map)
  • map_apply((k, v) -> struct(new_key, new_value), map)
  • transform_keys((k, v) -> new_key, map)
  • transform_values((k, v) -> new_value, map)

Example:

SELECT map_filter(
    (k, v) -> v > 10,
    map(1, 10, 2, 20)
);

SELECT transform_values(
    (k, v) -> v + 1,
    map(1, 10, 2, 20)
);

SELECT map_apply(
    (k, v) -> (k + 1, v * 2),
    map(1, 10, 2, 20)
);

Implementation

Use map_entries(m) to expand the map parameter and reuse the original array lambda for execution.

map_apply((k, v) -> (k2, v2), m)
    ↓
map_from_entries(
    array_apply(entry: struct{k, v} -> (k2, v2), map_entries(m)
);

map_filter((k, v) -> k + v > 0, m)
    ↓
%map_from_filtered_entries_unique%(
    array_map(
        entry: strct{k, v} -> if(
            e.k + e.v > 0, entry, null
        ),
    map_entries(m)
)


transform_values((k, v) -> k + v + col, m)
    ↓
%map_from_entries_unique%(
    array_map(
        entry -> (e.k, e.k + e.v + col),
        map_entries(m)
    )
)


map_exists((k, v) -> k + v = 1, m)
    ↓
array_match_any(
    array_map(
        entry -> e.k + e.v = 1,
        map_entries(m)
    )
)


map_all((k, v) -> k + v = 1, m)
    ↓
array_match_all(
    array_map(
        entry -> e.k + e.v = 1,
        map_entries(m)
    )
)

Internal helper function

This PR adds two internal Map construction functions used by the rewritten expressions:

  • %map_from_entries_unique%: rebuilds a Map from an entry array without running key deduplication. Used by transform_value
  • %map_from_filtered_entries_unique%: removes null entries input instead of throw error like map_from_entries. Used by map_filter

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66968

Problem Summary: Map offsets use 64-bit positions while IColumn::Selector stores 32-bit indexes. map_filter and the filtered-entry map constructor could therefore truncate a position at 2^32 and silently copy the wrong key and value. Validate the nested entry count before building a selector and return INVALID_ARGUMENT when an index cannot be represented. Add an O(1)-memory boundary unit test for the overflow case.

### Release note

Reject map operations whose nested positions exceed the selector range instead of returning corrupted data.

### Check List (For Author)

- Test: Unit Test (FunctionMapTest.* under ASAN)
- Behavior changed: Yes (oversized selector inputs now return INVALID_ARGUMENT)
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

Copy link
Copy Markdown
Contributor Author

run buildall

Issue Number: None

Related PR: apache#66968

Problem Summary: Higher-order function analysis selected map handling and array comparator behavior with hardcoded function-name checks. Move lambda parameter binding contracts into builtin function registration and use the resolved function builder metadata to analyze array and map lambdas through one flow.

None

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.nereids.rules.analysis.FunctionRegistryTest,org.apache.doris.nereids.trees.expressions.functions.scalar.MapLambdaFunctionsTest,org.apache.doris.nereids.rules.analysis.CheckExpressionLegalityTest,org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayFirstLastTest
    - mvn checkstyle:check -pl fe-core
- Behavior changed: No
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17096 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5fd8f3f5dff6648c0fd97bd879081f9904528885, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17565	3061	3018	3018
q2	2100	270	239	239
q3	10237	890	534	534
q4	4681	263	216	216
q5	7643	588	393	393
q6	139	119	96	96
q7	536	497	387	387
q8	9243	930	963	930
q9	3492	2423	2453	2423
q10	6528	892	708	708
q11	401	204	186	186
q12	616	272	201	201
q13	18110	1560	1162	1162
q14	158	161	148	148
q15	q16	440	394	374	374
q17	1411	852	731	731
q18	3137	2326	2298	2298
q19	1108	902	763	763
q20	358	284	204	204
q21	4858	1856	1909	1856
q22	334	275	229	229
Total cold run time: 93095 ms
Total hot run time: 17096 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3408	3345	3343	3343
q2	514	414	377	377
q3	2282	2343	2244	2244
q4	1196	1170	922	922
q5	2222	2152	2124	2124
q6	170	117	90	90
q7	1046	920	899	899
q8	1624	1431	1439	1431
q9	3173	3219	3150	3150
q10	1877	1823	1645	1645
q11	368	280	261	261
q12	461	451	347	347
q13	1506	1546	1172	1172
q14	182	169	175	169
q15	q16	403	406	366	366
q17	3644	3344	3268	3268
q18	4923	4518	4835	4518
q19	1005	886	899	886
q20	1042	972	838	838
q21	3943	3270	3259	3259
q22	400	355	325	325
Total cold run time: 35389 ms
Total hot run time: 31634 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 83883 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5fd8f3f5dff6648c0fd97bd879081f9904528885, data reload: false

query5	4292	426	328	328
query6	411	137	129	129
query7	4951	409	229	229
query8	294	136	123	123
query9	8676	3036	3033	3033
query10	404	214	177	177
query11	5378	1062	929	929
query12	124	74	80	74
query13	1217	459	310	310
query14	6018	2333	2212	2212
query14_1	2115	2099	2072	2072
query15	170	121	117	117
query16	923	401	386	386
query17	829	458	389	389
query18	2340	354	255	255
query19	188	145	112	112
query20	72	70	71	70
query21	204	103	89	89
query22	5404	5510	5431	5431
query23	6797	6301	6216	6216
query23_1	6418	6352	6036	6036
query24	7242	1094	763	763
query24_1	787	779	785	779
query25	422	290	238	238
query26	1221	226	125	125
query27	2800	406	259	259
query28	4722	1507	1510	1507
query29	981	427	355	355
query30	252	158	130	130
query31	862	417	340	340
query32	162	72	77	72
query33	455	224	185	185
query34	1003	822	480	480
query35	424	424	349	349
query36	569	567	555	555
query37	122	80	72	72
query38	1022	866	821	821
query39	503	494	515	494
query39_1	482	456	485	456
query40	201	94	79	79
query41	56	53	53	53
query42	76	72	76	72
query43	244	246	216	216
query44	1063	546	556	546
query45	112	107	101	101
query46	780	829	527	527
query47	777	787	701	701
query48	311	323	233	233
query49	543	249	190	190
query50	756	263	204	204
query51	8125	8248	8052	8052
query52	68	66	59	59
query53	197	228	196	196
query54	218	181	172	172
query55	78	60	56	56
query56	204	175	159	159
query57	698	660	663	660
query58	201	159	166	159
query59	1227	1235	1098	1098
query60	242	189	177	177
query61	148	143	138	138
query62	364	220	187	187
query63	171	144	165	144
query64	2986	840	727	727
query65	1606	1642	1642	1642
query66	1797	247	201	201
query67	9841	9958	9825	9825
query68	2768	1204	803	803
query69	348	221	197	197
query70	705	650	615	615
query71	253	182	176	176
query72	2456	1861	1629	1629
query73	617	590	338	338
query74	1589	1250	1142	1142
query75	1181	1128	979	979
query76	2286	766	543	543
query77	265	267	201	201
query78	4009	3675	3295	3295
query79	2312	849	599	599
query80	1456	337	288	288
query81	487	162	134	134
query82	922	144	103	103
query83	295	221	201	201
query84	306	115	89	89
query85	808	371	305	305
query86	405	180	178	178
query87	1033	974	914	914
query88	2783	2115	2122	2115
query89	294	199	180	180
query90	1966	127	136	127
query91	139	124	102	102
query92	85	66	75	66
query93	1390	1054	716	716
query94	671	277	218	218
query95	535	249	315	249
query96	781	596	269	269
query97	1135	1101	1095	1095
query98	149	144	142	142
query99	429	353	316	316
Total cold run time: 178607 ms
Total hot run time: 83883 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5fd8f3f5dff6648c0fd97bd879081f9904528885, data reload: false

query1	0.01	0.01	0.00
query2	0.08	0.04	0.04
query3	0.25	0.11	0.12
query4	1.61	0.09	0.10
query5	0.18	0.16	0.17
query6	1.24	0.72	0.73
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.23
query10	0.38	0.34	0.36
query11	0.17	0.11	0.12
query12	0.15	0.12	0.11
query13	0.31	0.31	0.32
query14	0.46	0.47	0.46
query15	0.37	0.35	0.36
query16	0.22	0.23	0.22
query17	0.73	0.72	0.70
query18	0.19	0.16	0.17
query19	1.16	1.28	1.24
query20	0.02	0.01	0.01
query21	15.46	0.16	0.11
query22	5.07	0.04	0.04
query23	16.18	0.27	0.10
query24	2.99	0.33	0.28
query25	0.11	0.04	0.03
query26	0.76	0.16	0.13
query27	0.04	0.04	0.03
query28	3.64	0.57	0.28
query29	12.46	3.18	2.59
query30	0.27	0.11	0.13
query31	2.77	0.38	0.18
query32	3.51	0.33	0.24
query33	1.39	1.42	1.43
query34	15.41	2.25	1.81
query35	1.77	1.79	1.76
query36	0.48	0.30	0.30
query37	0.06	0.04	0.03
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.12	0.08	0.07
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.62 s
Total hot run time: 15 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants