Support implicit lateral joins for set-returning functions in FROM

[zachmu]

Set-returning functions in the FROM list can now reference columns of preceding tables (implicit and explicit LATERAL joins).

Fixes #3112.

[itoqa]

Commit: b45b757: 14 test cases ran, 1 failed ❌, 12 passed ✅, 1 additional finding ⚠️.

Summary

Coverage spans correlated array expansion across common join forms, aliasing, row numbering, independent functions, and scope boundaries, including edge cases such as empty arrays and nested joins. The core query behavior is broadly healthy, but outer-join preservation has a correctness gap that can silently omit source data.

Merge with caution — this PR causes a medium-severity data-correctness failure in left joins when correlated expansion produces no rows, so affected results can be incomplete. A separate medium-severity parenthesized-join failure is unrelated to the PR and is a flag for later.

Tests run by Ito

Result	Severity	Type	Description
❌		Rev	The query returned only (2, 10) and (2, 20). It omitted the required (1, NULL) row even though the join was a LEFT JOIN.
✅	—	Alias	The query used the function name as its output name and returned rows (1,10), (1,20), and (2,30).
✅	—	General	Implicit and explicit lateral queries returned the right values for every outer row, even when the arrays had different lengths. The same associations stayed correct when a second table was added.
✅	—	General	Correlated array results keep the right values and numbering. Empty arrays return no rows, and numbering starts at 1 for each non-empty row in both query forms.
✅	—	General	Unqualified and schema-qualified unnest queries returned the same values. The explicit u(value) alias kept its renamed column and returned the correct rows for each table row.
✅	—	General	The correlated function returned the expected rows, while the neighboring subquery was rejected without an explicit LATERAL keyword. Adding LATERAL only to the subquery, or to both items, returned the expected rows.
✅	—	Function	The independent function returned 7 for both table rows, producing (1, 7) and (2, 7) as expected.
✅	—	Lateral	The query succeeded without the LATERAL keyword and returned the expected values for each row: (1,10), (1,20), and (2,30).
✅	—	Lateral	The query returned each array value beside the correct table row: (1,10), (1,20), and (2,30).
✅	—	Lateral	The query returned the expected rows for each input record, and the version with the optional keyword returned the same results.
✅	—	Ordinality	The query returned both array values for the first row and one value for the second row. Numbering started at 1 again for the second row, as expected.
✅	—	Ordinality	The implicit query and the query with CROSS JOIN LATERAL returned the same values and row numbers for both input arrays.
✅	—	Subquery	A subquery in the FROM list could not read a table placed before it unless the query used the explicit LATERAL keyword. The explicit form succeeded, so function handling did not widen subquery scope.
⚠️		General	The query stops with an internal unsupported-expression error when the joined tables are wrapped in parentheses. The expected unnested values are never returned.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Parenthesized joins fail before scope checking

• Severity: Medium
• Description: The query stops with an internal unsupported-expression error when the joined tables are wrapped in parentheses. The expected unnested values are never returned.
• Impact: Queries that use a parenthesized joined table with array unnesting fail instead of returning results. Users can usually rewrite the query without the parentheses, but the affected query form does not work.
• Steps to Reproduce:
  1. Create local tables t1(id integer, arr integer[]) and t2(id integer, name text), with t1 rows (1, ARRAY[10,20]) and (2, ARRAY[30]) and matching t2 rows.
  2. Run SELECT j.id, u.value FROM (t1 JOIN t2 ON t1.id=t2.id) AS j CROSS JOIN unnest(j.arr) AS u(value) ORDER BY j.id, u.value.
  3. Observe the error unhandled table expression: *tree.ParenTableExpr instead of rows for ids 1 and 2.
• Stub / mock content: The query used local test tables and array rows created for this verification; no stubs, mocks, route interception, or bypasses were applied.
• Code Analysis: In server/ast/select_clause.go:112-117, the PR calls markImplicitLateralFunctions before nodeFrom. Its recursive branch at lines 184-190 correctly walks a JoinTableExpr and a ParenTableExpr, but that only annotates the parsed tree. Conversion then reaches nodeAliasedTableExpr in server/ast/aliased_table_expr.go. That function handles TableName, Subquery, and RowsFromExpr at lines 40-161; its default at lines 162-164 returns errors.Errorf("unhandled table expression: %T", expr). For the query's AS j wrapper, node.Expr is *tree.ParenTableExpr, so conversion exits at that default before the later lateral rewrite can run. server/ast/table_expr.go:93-100 supports ParenTableExpr only when it is converted as a top-level table expression, not when it is nested inside AliasedTableExpr.Expr. The smallest practical fix is to add a ParenTableExpr case to nodeAliasedTableExpr that converts the inner join and preserves the alias, or otherwise unwraps this specific wrapper before the existing table-expression conversion. The PR diff does not change this function, so this is a pre-existing conversion gap exposed by the test shape rather than a regression caused by the PR.

Evidence Package

• text/BF-ORDER-1/BF-ORDER-1-parenthesized-join-evidence.txt — 3 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Empty lateral results drop source rows

What failed: The query returned only (2, 10) and (2, 20). It omitted the required (1, NULL) row even though the join was a LEFT JOIN.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug

• Severity: Medium
• Impact: Queries using a left lateral join can silently omit source rows when the function returns no values. Reports and data processing that rely on those results may be incomplete.
• Steps to Reproduce:
  1. Create a table with one row containing an empty integer array and another row containing [10, 20].
  2. Run SELECT t.id, u FROM t LEFT JOIN LATERAL unnest(t.values) AS x(u) ON true ORDER BY t.id, u NULLS FIRST.
  3. Check that the empty-array row is returned with a NULL function value, alongside the two values from the non-empty row.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: In server/ast/select_clause.go, nodeSelectClause calls markImplicitLateralFunctions before converting the FROM clause (lines 112-117). That PR-added helper marks a following RowsFromExpr as lateral (lines 170-194). Later, rewriteTableFuncExprs recognizes the function wrapper and, when expr.Lateral is true, converts the TableFuncExpr into an AliasedTableExpr containing a lateral subquery (lines 242-260). This makes the preceding table visible to the function, but the resulting execution path does not preserve the outer join's null-extended row when the table function produces zero rows. The smallest practical fix is in this lateral conversion path: preserve LEFT JOIN semantics for an empty TableFuncExpr result, either by representing the lateral function in a form that the existing join executor null-extends or by adding the narrow null-extension handling at this wrapper boundary; do not change unrelated non-lateral function or subquery behavior.
• Why this is likely a bug: This is a real local Doltgres execution failure, not a harness-only symptom: the SQL command succeeded and consistently returned two rows instead of the three rows required by PostgreSQL outer-join semantics. The fixture uses ordinary integer arrays and the recorded setup did not stub or intercept the database behavior. The PR changed the exact AST conversion path used by the query, and the missing row is silently lost rather than reported as an unsupported query. A targeted fix to the new lateral wrapper or its join representation should restore null extension while retaining the PR's intended correlated-function support.

Relevant code

server/ast/select_clause.go:112-117

// We mirror that here by marking any function-call FROM item that follows another FROM item as lateral before
// converting the FROM clause.
markImplicitLateralFunctions(node.From.Tables)
from, err := nodeFrom(ctx, node.From)

server/ast/select_clause.go:242-260

tableFuncExpr := &vitess.TableFuncExpr{
	Name: funcExpr.Name.String(),
	Exprs: funcExpr.Exprs,
	Alias: alias,
	Columns: subquery.Columns,
}
if expr.Lateral {
	return &vitess.AliasedTableExpr{
		Expr: &vitess.Subquery{
			Select: &vitess.Select{
				SelectExprs: vitess.SelectExprs{&vitess.StarExpr{}},
				From: vitess.TableExprs{tableFuncExpr},
			},
		},
		As: alias,
		Lateral: true,
	}

server/ast/select_clause.go:170-181

// markImplicitLateralFunctions marks function-call FROM items that follow another FROM item as lateral.
func markImplicitLateralFunctions(tables tree.TableExprs) {
	var mark func(table tree.TableExpr, followsFromItem bool)
	mark = func(table tree.TableExpr, followsFromItem bool) {
		switch table := table.(type) {
		case *tree.AliasedTableExpr:
			if followsFromItem {
				if _, ok := table.Expr.(*tree.RowsFromExpr); ok {
					table.Lateral = true

Evidence Package

• text/REV-2/REV-2-left-join-lateral.txt — 2 KB

Copy prompt for an agent

Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**Medium severity — Empty lateral results drop source rows**

**What failed:** The query returned only (2, 10) and (2, 20). It omitted the required (1, NULL) row even though the join was a LEFT JOIN.

- **Impact:** Queries using a left lateral join can silently omit source rows when the function returns no values. Reports and data processing that rely on those results may be incomplete.
- **Steps to reproduce:**
  1. Create a table with one row containing an empty integer array and another row containing [10, 20].
  2. Run SELECT t.id, u FROM t LEFT JOIN LATERAL unnest(t.values) AS x(u) ON true ORDER BY t.id, u NULLS FIRST.
  3. Check that the empty-array row is returned with a NULL function value, alongside the two values from the non-empty row.
- **Stub / mock content:** No stubs, mocks, or bypasses were applied for this test in the recorded run.
- **Code analysis:** In server/ast/select_clause.go, nodeSelectClause calls markImplicitLateralFunctions before converting the FROM clause (lines 112-117). That PR-added helper marks a following RowsFromExpr as lateral (lines 170-194). Later, rewriteTableFuncExprs recognizes the function wrapper and, when expr.Lateral is true, converts the TableFuncExpr into an AliasedTableExpr containing a lateral subquery (lines 242-260). This makes the preceding table visible to the function, but the resulting execution path does not preserve the outer join's null-extended row when the table function produces zero rows. The smallest practical fix is in this lateral conversion path: preserve LEFT JOIN semantics for an empty TableFuncExpr result, either by representing the lateral function in a form that the existing join executor null-extends or by adding the narrow null-extension handling at this wrapper boundary; do not change unrelated non-lateral function or subquery behavior.
- **Why this is likely a bug:** This is a real local Doltgres execution failure, not a harness-only symptom: the SQL command succeeded and consistently returned two rows instead of the three rows required by PostgreSQL outer-join semantics. The fixture uses ordinary integer arrays and the recorded setup did not stub or intercept the database behavior. The PR changed the exact AST conversion path used by the query, and the missing row is silently lost rather than reported as an unsupported query. A targeted fix to the new lateral wrapper or its join representation should restore null extension while retaining the PR's intended correlated-function support.

**Relevant code:**

`server/ast/select_clause.go:112-117`

~~~go
// We mirror that here by marking any function-call FROM item that follows another FROM item as lateral before
// converting the FROM clause.
markImplicitLateralFunctions(node.From.Tables)
from, err := nodeFrom(ctx, node.From)
~~~

`server/ast/select_clause.go:242-260`

~~~go
tableFuncExpr := &vitess.TableFuncExpr{
	Name: funcExpr.Name.String(),
	Exprs: funcExpr.Exprs,
	Alias: alias,
	Columns: subquery.Columns,
}
if expr.Lateral {
	return &vitess.AliasedTableExpr{
		Expr: &vitess.Subquery{
			Select: &vitess.Select{
				SelectExprs: vitess.SelectExprs{&vitess.StarExpr{}},
				From: vitess.TableExprs{tableFuncExpr},
			},
		},
		As: alias,
		Lateral: true,
	}
~~~

`server/ast/select_clause.go:170-181`

~~~go
// markImplicitLateralFunctions marks function-call FROM items that follow another FROM item as lateral.
func markImplicitLateralFunctions(tables tree.TableExprs) {
	var mark func(table tree.TableExpr, followsFromItem bool)
	mark = func(table tree.TableExpr, followsFromItem bool) {
		switch table := table.(type) {
		case *tree.AliasedTableExpr:
			if followsFromItem {
				if _, ok := table.Expr.(*tree.RowsFromExpr); ok {
					table.Lateral = true
~~~

[github-actions]

	Main	PR
Total	42090	42090
Successful	18981	18990
Failures	23109	23100
Partial Successes1	5461	5463

	Main	PR
Successful	45.0962%	45.1176%
Failures	54.9038%	54.8824%

${\color{red}Regressions (1)}$

subselect

QUERY:          select count(*) from tenk1 t
where (exists(select 1 from tenk1 k where k.unique1 = t.unique2) or ten < 0);
RECEIVED ERROR: timeout during Receive

${\color{lightgreen}Progressions (10)}$

join

QUERY: select * from
(values (1, array[10,20]), (2, array[20,30])) as v1(v1x,v1ys)
left join (values (1, 10), (2, 20)) as v2(v2x,v2y) on v2x = v1x
left join unnest(v1ys) as u1(u1y) on u1y = v2y;

QUERY: select count(*) from tenk1 a, lateral generate_series(1,two) g;

QUERY: select * from (values(1)) x(lb),
  lateral generate_series(lb,4) x4;

QUERY: select * from (select f1/1000000000 from int4_tbl) x(lb),
  lateral generate_series(lb,4) x4;

rangefuncs

QUERY: SELECT * FROM (VALUES (1),(2),(3)) v(r), generate_series(10+r,20-r) f(i);

QUERY: SELECT * FROM (VALUES (1),(2),(3)) v(r), generate_series(10+r,20-r) WITH ORDINALITY AS f(i,o);

QUERY: SELECT * FROM (VALUES (1),(2),(3)) v(r), unnest(array[r*10,r*20,r*30]) f(i);

QUERY: SELECT * FROM (VALUES (1),(2),(3)) v(r), unnest(array[r*10,r*20,r*30]) WITH ORDINALITY AS f(i,o);

QUERY: SELECT * FROM (VALUES (1),(2),(3)) v1(r1),
              LATERAL (SELECT r1, * FROM (VALUES (10),(20),(30)) v2(r2)
                                         LEFT JOIN generate_series(r2,r2+3) f(i) ON ((r2+i)<100) OFFSET 0) s1;

QUERY: SELECT * FROM (VALUES (1),(2),(3)) v1(r1),
              LATERAL (SELECT r1, * FROM (VALUES (10),(20),(30)) v2(r2)
                                         LEFT JOIN generate_series(r1,2+r2/5) f(i) ON ((r2+i)<100) OFFSET 0) s1;

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[coffeegoddd]

@zachmu DOLT

read_tests	from_latency_median	to_latency_median	is_faster
covering_index_scan_postgres	2.43	2.48	0
groupby_scan_postgres	75.82	75.82	0
index_join_postgres	2.18	2.22	0
index_join_scan_postgres	1.58	1.58	0
index_scan_postgres	484.44	493.24	0
oltp_point_select	0.36	0.36	0
oltp_read_only	6.32	6.32	0
select_random_points	0.7	0.7	0
select_random_ranges	1.01	1.01	0
table_scan_postgres	484.44	475.79	0
types_table_scan_postgres	1213.57	1213.57	0

write_tests	from_latency_median	to_latency_median	is_faster
bulk_insert	0.001	0.001	0
oltp_delete_insert_postgres	6.67	6.67	0
oltp_insert	3.36	3.36	0
oltp_read_write	13.22	13.22	0
oltp_update_index	3.55	3.55	0
oltp_update_non_index	3.25	3.25	0
oltp_write_only	6.91	6.91	0
types_delete_insert_postgres	7.17	7.17	0