[SPARK-53154][SQL][PYTHON] Add hmac SQL function#57344
Open
vinodkc wants to merge 2 commits into
Open
Conversation
Add a built-in `hmac(key, message[, algorithm])` scalar function across SQL, the Scala DataFrame API, and PySpark (classic + Connect). Returns raw MAC bytes (BinaryType) so results can be chained as keys; default algorithm SHA-256. Co-authored-by: Isaac
hmac SQL functionhmac SQL function
Ma77Ball
reviewed
Jul 18, 2026
The crypto error (empty/invalid key) can only originate from the `key` argument, not `message`, so listing both was misleading.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a built-in
hmac(key, message[, algorithm])scalar function that computes a keyed-hash message authentication code (HMAC) using the JVM'sjavax.crypto.Mac.API surface added:
hmac(key, message)/hmac(key, message, algorithm)functions.hmac(key, message)/functions.hmac(key, message, algorithm)pyspark.sql.functions.hmac(key, message, algorithm=None)Key design choices:
BinaryType(raw MAC bytes), not a hex string — so the output of onehmaccall feeds directly into thekeyof the next with nounhexin between.This is the critical property for chained key-derivation (e.g. AWS SigV4).
(key, message), matching Python'shmac.new(key, msg)and RFC 2104'sHMAC(K, m).SHA-256andSHA256spellings accepted).Why are the changes needed?
Spark had no single expression node for HMAC. Users had to hand-build the RFC 2104
formula from
sha2,concat, bitwisexor, andunhexprimitives. The motivatingreal-world use case derives an AWS SigV4 signing key by chaining four HMACs, each step's output becoming
the next step's key:
Without this function, each HMAC step is a deep sha2/unhex/concat/xor subtree.
Nesting four of them multiplies the tree size to the point where the Catalyst analyzer/optimizer runs out of memory on realistic inputs. hmac collapses each level to a single StaticInvoke node.
Does this PR introduce any user-facing change?
Yes. Adds a new built-in SQL function
hmacand corresponding Scala/PySparkDataFrame API entries. No existing behavior is changed.
Example:
How was this patch tested?
Added tests in
ExpressionImplUtilsSuite,MiscExpressionsSuite,misc-functions.sqlWas this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (sonnet-5)