feat: [branch-1.0] add codegen dispatch fallback for CometCast incompatible/unsupported cases (#5079)

[andygrove]

Which issue does this PR close?

Cherry-pick of #5079 (merged to main as 76b9d2eee) onto branch-1.0.

Rationale for this change

CometCast was the last serde on the incompatible/unsupported path that had no codegen dispatch. Without it, a single cast that Comet declines to run natively forces the whole projection back to Spark, so the surrounding native operators are lost too. The cases affected are all reachable with default or common configuration:

• spark.sql.legacy.castComplexTypesToString.enabled=true (struct/array/map to string) was Unsupported, forcing full fallback.
• Non-CORRECTED spark.sql.legacy.timeParserPolicy (LEGACY, EXCEPTION) on string to date/timestamp casts.
• MapType to string, and struct/array containing a MapType, which have no native implementation.
• Pre-existing Incompatible cases (negative-scale decimal to string, float/double to decimal rounding).

branch-1.0 already carries CodegenDispatchFallback and the serdes that use it, so this is the same mechanism already shipping for In/InSet, strings, datetime, arrays, and maps — CometCast was simply missed. Getting it into 1.0.0 avoids a release where a legacy cast config silently costs users native execution of an entire projection.

What changes are included in this PR?

A git cherry-pick -x of 76b9d2eee. The content is identical to the commit on main; the two dev/diffs/*.diff hunks auto-merged (only surrounding comet.version context differs on this branch, since branch-1.0 is at 1.0.0 rather than 1.1.0-SNAPSHOT), and no adaptation was needed.

• CometCast mixes in CodegenDispatchFallback, so casts it cannot match Spark bit-for-bit run Spark's own doGenCode inside the Comet pipeline instead of falling the projection back.
• spark.sql.legacy.castComplexTypesToString.enabled=true now routes through the dispatcher; the complex-type check is deduplicated against DataTypeSupport.isComplexType, and the SQLConf lookup is reordered to run only when the source type is complex.
• VariantType (Spark 4, SPARK-45827) is rejected in CometCast.getSupportLevel / isSupported via a new isVariantType in CometTypeShim (stubbed false on Spark 3.x). Neither the native serde nor the codegen dispatcher can carry VariantType in the data args or return type, so these casts must reach Spark. The guard also covers the Literal short-circuit branch, where constant folding would otherwise produce a Literal[VariantType] that no downstream serde can serialize.
• Tests: expect_fallback markers dropped from the non-legacy cast_complex_types_to_string cases that now stay in the Comet pipeline, plus a missing query directive restored above the array-of-map case (it was previously an orphan SELECT that never ran); CometCastSuite checks the Comet operator rather than a Spark fallback reason for ArrayType(DateType) to an unsupported ArrayType; CometExpressionSuite's explain comet expectations updated for the CalendarIntervalType to string cast, which no longer emits a fallback reason.
• dev/diffs/4.0.2.diff and 4.1.2.diff: loosen VariantEndToEndSuite's codegen-mode plan-type assertion to also accept CometNativeColumnarToRowExec, since under Comet the plan is wrapped rather than having WholeStageCodegenExec at the top.

How are these changes tested?

By the tests carried in the cherry-pick. Verified locally on this branch with a debug native build:

• CometSqlFileTestSuite filtered to cast_complex_types_to_string — 3 tests pass (legacy file, plus non-legacy with dictionary encoding on and off).
• CometCastSuite — 163 tests pass on the default profile, and 162 pass (1 canceled) under -Pspark-3.5, which exercises the spark-3.x isVariantType stub.
• CometExpressionSuite — 138 tests pass, covering the updated explain comet expectations.
• ./mvnw spotless:check scalastyle:check clean on comet-spark.
• Both updated Spark diffs verified with git apply --check against v4.0.2 and v4.1.2 checkouts.