feat: add comet_version() SQL function

[andygrove]

Which issue does this PR close?

Rationale for this change

There is currently no built-in way for a Spark user to programmatically query which Comet version is loaded. The JVM COMET_VERSION constant exists in org.apache.comet.package but is only accessible via Scala imports; the native library also logs the version on init, but that requires driver log access. This PR exposes the version as a plain SQL function so it is reachable from PySpark, spark-sql, notebooks, and any other client that can issue SQL — no JVM/Scala imports and no log spelunking required.

What changes are included in this PR?

• Register a nullary SQL function comet_version() via SparkSessionExtensions.injectFunction in CometSparkSessionExtensions. The function returns the COMET_VERSION string (already populated from comet-git-info.properties at build time) as a Spark StringType literal.
• Reject any arguments with an IllegalArgumentException mentioning the function name.
• Add two unit tests in CometSparkSessionExtensionsSuite: one asserting SELECT comet_version() returns the expected build version, and one asserting comet_version(1) fails.

Compiles cleanly against spark-3.4, spark-3.5, spark-4.0, and spark-4.1 profiles.

Example usage:

scala> spark.sql("SELECT comet_version()").show(false)
+----------------+
|comet_version() |
+----------------+
|1.0.0-SNAPSHOT  |
+----------------+

How are these changes tested?

• New unit tests in CometSparkSessionExtensionsSuite (comet_version() returns the Comet build version and comet_version() rejects arguments).
• Full suite runs green locally with the spark-3.5 profile.
• Verified compilation with -Pspark-3.4, -Pspark-3.5, and -Pspark-4.1 -Pscala-2.13.

[comphead]

Thanks @andygrove do we really need a function, we can come up with constants like sparkVerson on the sparkContext. Especially it would be more useful if we plan to expose Comet version to some sort of UI.

Currently it is possible to know Comet version by:

• checking classpath in spark UI
• asking loaded comet plugin jar filename

[andygrove]

Thanks @andygrove do we really need a function, we can come up with constants like sparkVerson on the sparkContext. Especially it would be more useful if we plan to expose Comet version to some sort of UI.

Currently it is possible to know Comet version by:

• checking classpath in spark UI
• asking loaded comet plugin jar filename

Today, users can evaluate println(org.apache.comet.COMET_VERSION) in Scala spark-shell. I thought that having a SQL function would be more intuitive though.

[andygrove]

Thanks @andygrove do we really need a function, we can come up with constants like sparkVerson on the sparkContext. Especially it would be more useful if we plan to expose Comet version to some sort of UI.
Currently it is possible to know Comet version by:

• checking classpath in spark UI
• asking loaded comet plugin jar filename

Today, users can evaluate println(org.apache.comet.COMET_VERSION) in Scala spark-shell. I thought that having a SQL function would be more intuitive though.

The PySpark version is less intuitive:

jvm = spark.sparkContext._jvm
pkg = getattr(jvm.org.apache.comet, "package$")
print(getattr(pkg, "MODULE$").COMET_VERSION())

[andygrove]

@comphead Bumping this — the previous reply covered why a function-based path is complementary to the classpath / plugin-jar inspection you outlined. If you would rather I add a `SparkSessionExtensions`-registered constant (e.g. `spark.sql.function.comet_version = `) instead of or in addition to the SQL function, happy to do that.

[comphead]

@comphead Bumping this — the previous reply covered why a function-based path is complementary to the classpath / plugin-jar inspection you outlined. If you would rather I add a SparkSessionExtensions-registered constant (e.g. spark.sql.function.comet_version = ) instead of or in addition to the SQL function, happy to do that.

Thanks @andygrove the function is also great, however it would bring up insignificant maintenance burden. If we follow spark then registered constant would be more expected way. Actually I'm fine either ways, leaving the final decision to you.

The only thing is whatever way, we prob should describe it in User Guide or plugin_overview.md

[comphead]

Thanks @andygrove CI pending

[andygrove]

Closing in favor of #5049, which exposes the version as the spark.comet.version runtime config instead of a SQL function. Per review feedback here, a config avoids polluting the function registry (which also broke the upstream ShowFunctionsSuite tests) and is queryable via spark.conf.get / SET.