Add support for CTE operators

[jianhongshi123]

Materialized CTE operators do not require special handling because the default case recursively visits their children. However, CTE references were not registered in the transfer DAG.

This PR registers CTE references as separate transfer DAG nodes.

[JP-Reddy]

So I ran the query and I see our operators are still missing from the plan. The join edges for the CTE tables are missing and the optimizer skips over DAG creation since no. of edges is under 2.

[jianhongshi123]

If the query you refer to is q15, I think it actually contains only one join on s_suppkey = supplier_no between supplier and revenue, so not inserting CREATE/PROBE is consistent with our current design.

[JP-Reddy]

The query has two hash joins:

1. supplier.s_suppkey = revenue.supplier_no
2. total_revenue = SUBQUERY

we may not see improvements for this query since the aggregate emits just one row, but this could be beneficial for queries with group bys.

┌─────────────────────────────┐
│┌───────────────────────────┐│
││       Physical Plan       ││
│└───────────────────────────┘│
└─────────────────────────────┘
┌───────────────────────────┐
│            CTE            │
│    ────────────────────   │
│     CTE Name: revenue     │
│       Table Index: 0      ├──────────────┐
│                           │              │
│          ~3 rows          │              │
└─────────────┬─────────────┘              │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│         PROJECTION        ││         PROJECTION        │
│    ────────────────────   ││    ────────────────────   │
│__internal_decompress_integ││__internal_decompress_integ│
│     ral_bigint(#0, 1)     ││     ral_bigint(#0, 1)     │
│             #1            ││             #1            │
│                           ││             #2            │
│                           ││__internal_decompress_strin│
│                           ││           g(#3)           │
│                           ││             #4            │
│                           ││                           │
│        ~8,565 rows        ││          ~3 rows          │
└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│       HASH_GROUP_BY       ││          ORDER_BY         │
│    ────────────────────   ││    ────────────────────   │
│         Groups: #0        ││   tpch_sf1.main.supplier  │
│                           ││       .s_suppkey ASC      │
│        Aggregates:        ││                           │
│    sum_no_overflow(#1)    ││                           │
│                           ││                           │
│        ~8,565 rows        ││                           │
└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│         PROJECTION        ││         PROJECTION        │
│    ────────────────────   ││    ────────────────────   │
│        supplier_no        ││__internal_compress_integra│
│ (l_extendedprice * (1.00 -││     l_usmallint(#0, 1)    │
│        l_discount))       ││             #1            │
│                           ││             #2            │
│                           ││__internal_compress_string_│
│                           ││        uhugeint(#3)       │
│                           ││             #4            │
│                           ││                           │
│      ~1,200,243 rows      ││          ~3 rows          │
└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│         PROJECTION        ││         HASH_JOIN         │
│    ────────────────────   ││    ────────────────────   │
│__internal_compress_integra││      Join Type: INNER     │
│     l_usmallint(#0, 1)    ││                           │
│             #1            ││        Conditions:        ├──────────────┐
│             #2            ││  s_suppkey = supplier_no  │              │
│                           ││                           │              │
│      ~1,200,243 rows      ││          ~3 rows          │              │
└─────────────┬─────────────┘└─────────────┬─────────────┘              │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│          SEQ_SCAN         ││          SEQ_SCAN         ││         HASH_JOIN         │
│    ────────────────────   ││    ────────────────────   ││    ────────────────────   │
│           Table:          ││           Table:          ││      Join Type: INNER     │
│   tpch_sf1.main.lineitem  ││   tpch_sf1.main.supplier  ││                           │
│                           ││                           ││        Conditions:        │
│   Type: Sequential Scan   ││   Type: Sequential Scan   ││  total_revenue = SUBQUERY │
│                           ││                           ││                           │
│        Projections:       ││        Projections:       ││                           │
│         l_suppkey         ││         s_suppkey         ││                           │
│      l_extendedprice      ││           s_name          ││                           ├──────────────┐
│         l_discount        ││         s_address         ││                           │              │
│                           ││          s_phone          ││                           │              │
│          Filters:         ││                           ││                           │              │
│ l_shipdate>='1996-01-01': ││                           ││                           │              │
│ :DATE AND l_shipdate<'1996││                           ││                           │              │
│       -04-01'::DATE       ││                           ││                           │              │
│                           ││                           ││                           │              │
│      ~1,200,243 rows      ││        ~10,000 rows       ││          ~3 rows          │              │
└───────────────────────────┘└───────────────────────────┘└─────────────┬─────────────┘              │
                                                          ┌─────────────┴─────────────┐┌─────────────┴─────────────┐
                                                          │          CTE_SCAN         ││         PROJECTION        │
                                                          │    ────────────────────   ││    ────────────────────   │
                                                          │        CTE Index: 0       ││ CASE  WHEN ((#1 > 1)) THEN│
                                                          │                           ││   ("error"('More than one │
                                                          │                           ││      row returned by a    │
                                                          │                           ││     subquery used as an   │
                                                          │                           ││     expression - scalar   │
                                                          │                           ││     subqueries can only   │
                                                          │                           ││    return a single row.   │
                                                          │                           ││          Use "SET         │
                                                          │                           ││ scalar_subquery_error_on_m│
                                                          │                           ││   ultiple_rows=false" to  │
                                                          │                           ││     revert to previous    │
                                                          │                           ││   behavior of returning a │
                                                          │                           ││ random row.')) ELSE #0 END│
                                                          │                           ││                           │
                                                          │        ~8,565 rows        ││           ~1 row          │
                                                          └───────────────────────────┘└─────────────┬─────────────┘
                                                                                       ┌─────────────┴─────────────┐
                                                                                       │    UNGROUPED_AGGREGATE    │
                                                                                       │    ────────────────────   │
                                                                                       │        Aggregates:        │
                                                                                       │        "first"(#0)        │
                                                                                       │        count_star()       │
                                                                                       └─────────────┬─────────────┘
                                                                                       ┌─────────────┴─────────────┐
                                                                                       │         PROJECTION        │
                                                                                       │    ────────────────────   │
                                                                                       │             #0            │
                                                                                       │                           │
                                                                                       │           ~1 row          │
                                                                                       └─────────────┬─────────────┘
                                                                                       ┌─────────────┴─────────────┐
                                                                                       │    UNGROUPED_AGGREGATE    │
                                                                                       │    ────────────────────   │
                                                                                       │    Aggregates: max(#0)    │
                                                                                       └─────────────┬─────────────┘
                                                                                       ┌─────────────┴─────────────┐
                                                                                       │         PROJECTION        │
                                                                                       │    ────────────────────   │
                                                                                       │       total_revenue       │
                                                                                       │                           │
                                                                                       │        ~8,565 rows        │
                                                                                       └─────────────┬─────────────┘
                                                                                       ┌─────────────┴─────────────┐
                                                                                       │          CTE_SCAN         │
                                                                                       │    ────────────────────   │
                                                                                       │        CTE Index: 0       │
                                                                                       │                           │
                                                                                       │        ~8,565 rows        │
                                                                                       └───────────────────────────┘

[jianhongshi123]

You're right, there are actually two hash joins but only one join edge extracted. After tracing the second join that is not turned into join edge, I found that the right child contains ungrouped aggregate, so the binding can not be mapped to cte scan in our current code. However, the binding tracing can go across grouped aggregates, which means join edges can be extracted if the aggregate here is group by.

[JP-Reddy]

okay that's good, then. Thanks!