Skip to content

[SPARK-53154][SQL][PYTHON] Add hmac SQL function#57344

Open
vinodkc wants to merge 2 commits into
apache:masterfrom
vinodkc:br_SPARK-53154
Open

[SPARK-53154][SQL][PYTHON] Add hmac SQL function#57344
vinodkc wants to merge 2 commits into
apache:masterfrom
vinodkc:br_SPARK-53154

Conversation

@vinodkc

@vinodkc vinodkc commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add a built-in hmac(key, message[, algorithm]) scalar function that computes a keyed-hash message authentication code (HMAC) using the JVM's javax.crypto.Mac.

API surface added:

  • SQL: hmac(key, message) / hmac(key, message, algorithm)
  • Scala DataFrame: functions.hmac(key, message) / functions.hmac(key, message, algorithm)
  • PySpark (classic + Connect): pyspark.sql.functions.hmac(key, message, algorithm=None)

Key design choices:

  • Returns BinaryType (raw MAC bytes), not a hex string — so the output of one
    hmac call feeds directly into the key of the next with no unhex in between.
    This is the critical property for chained key-derivation (e.g. AWS SigV4).
  • Argument order is (key, message), matching Python's hmac.new(key, msg) and RFC 2104's HMAC(K, m).
  • Default algorithm is SHA-256. Supported: SHA-224, SHA-256, SHA-384, SHA-512, SHA-1, MD5 (case-insensitive; both SHA-256 and SHA256 spellings accepted).

Why are the changes needed?

Spark had no single expression node for HMAC. Users had to hand-build the RFC 2104
formula from sha2, concat, bitwise xor, and unhex primitives. The motivating
real-world use case derives an AWS SigV4 signing key by chaining four HMACs, each step's output becoming
the next step's key:

-- Four-step SigV4 key derivation (AWS docs, "Signature Version 4").
-- Each hmac() output is BinaryType, feeding directly as the next key -- no unhex anywhere.
SELECT hex(
  hmac(
    hmac(
      hmac(
        hmac(concat('AWS4', secret), date),
        region),
      service),
    'aws4_request')
) AS signing_key;

Without this function, each HMAC step is a deep sha2/unhex/concat/xor subtree.
Nesting four of them multiplies the tree size to the point where the Catalyst analyzer/optimizer runs out of memory on realistic inputs. hmac collapses each level to a single StaticInvoke node.

Does this PR introduce any user-facing change?

Yes. Adds a new built-in SQL function hmac and corresponding Scala/PySpark
DataFrame API entries. No existing behavior is changed.

Example:

-- Default algorithm (SHA-256)
SELECT hex(hmac('key', 'message'));
-- 6E9EF29B75FFFC5B7ABAE527D58FDADB2FE42E7219011976917343065F58ED4A
-- Explicit algorithm
SELECT hex(hmac('key', 'message', 'SHA-512'));
-- E477384D7CA229DD1426E64B63EBF2D36EBD6D7E669A6735424E72EA6C01D3F8B56EB39C36D8232F5427999B8D1A3F9CD1128FC69F4D75B434216810FA367E98
-- Chaining: binary output feeds directly as the next key
SELECT hex(hmac(hmac('key', 'message'), 'message2'));
-- 28042756E0362D954B61661BB4DEC9FF9F6C970D346CAEB6DB36ED7B735AB38C

How was this patch tested?

Added tests in ExpressionImplUtilsSuite, MiscExpressionsSuite , misc-functions.sql

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (sonnet-5)

Add a built-in `hmac(key, message[, algorithm])` scalar function across SQL,
the Scala DataFrame API, and PySpark (classic + Connect). Returns raw MAC bytes
(BinaryType) so results can be chained as keys; default algorithm SHA-256.

Co-authored-by: Isaac
@vinodkc vinodkc changed the title [SPARK-53154][SQL][PYTHON] Add native hmac SQL function [SPARK-53154][SQL][PYTHON] Add hmac SQL function Jul 17, 2026

@Ma77Ball Ma77Ball left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, adding an HMAC function sounds like a great addition. I left two minor optional nits below.

The crypto error (empty/invalid key) can only originate from the `key`
argument, not `message`, so listing both was misleading.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants