Skip to content

feat: explode(map) - #8235

Open
betodealmeida wants to merge 3 commits into
tobymao:mainfrom
betodealmeida:spark-explode
Open

feat: explode(map)#8235
betodealmeida wants to merge 3 commits into
tobymao:mainfrom
betodealmeida:spark-explode

Conversation

@betodealmeida

Copy link
Copy Markdown
Contributor

Introduce a kind="map" for the Explode expressions; this is needed because in Spark we can have:

-- spark
SELECT
  EXPLODE(map_column) AS (key, value)
FROM t

Normally this gets transpiled to an array unnest. For example, when transpiled to Trino:

>>> import sqlglot
>>> print(sqlglot.parse_one("SELECT EXPLODE(map_column) AS (key, value) FROM t", "spark2").sql("trino", pr\
etty=True))
SELECT
  IF(_u.pos = _u_2.key, _u_2.value) AS value
FROM t
CROSS JOIN UNNEST(SEQUENCE(1, GREATEST(CARDINALITY(map_column)))) AS _u(pos)
CROSS JOIN UNNEST(map_column) WITH ORDINALITY AS _u_2(value, key)
WHERE
  _u.pos = _u_2.key
  OR (
    _u.pos > CARDINALITY(map_column) AND _u_2.key = CARDINALITY(map_column)
  )
>>>

The query above is invalid here:

CROSS JOIN UNNEST(map_column) WITH ORDINALITY AS _u_2(value, key)

Since in Trino UNNEST(map_column) with ORDINALITY applied to a map returns 3 columns (key, value, index), not the expected 2.

With the changes in this PR, the parser assumes that an EXPLODE() followed by 2 aliases is being applied on a map. It then sets kind="map" on the Explode, which gets rendered correctly by Trino as:

>>> import sqlglot
>>> print(sqlglot.parse_one("SELECT EXPLODE(map_column) AS (key, value) FROM t", "spark2").sql("trino", pretty=True))
SELECT
  _u_2.key AS key,
  _u_2.value AS value
FROM t
CROSS JOIN UNNEST(map_column) AS _u_2(key, value)
>>>

Comment thread sqlglot/parsers/spark2.py
and isinstance(this.this, exp.Explode)
and not isinstance(this.this, (exp.Posexplode, exp.ExplodeOuter))
):
this.this.set("kind", "map")

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think this is the right approach, in order for this to work we need to know the type. so we should actually use type annotations. get rid of map as an argument, instead set the data type of the expression. that way even without the alias, if we use the type annotator, this still can work

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you check that this suggestion works across the various dialects that have Explode?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants