Revisit format/transform interface and implementation (datasets 4 lazy Columns)

[CarloLucibello]

Summary

Revisit the format / transform interface (and its implementation) now that
datasets >= 4 returns lazy Column objects for column access. The current
behavior is intentional and documented — this issue is a placeholder to
reconsider the design, not a bug report.

Current model

A Dataset has two independent underlying slots:

• the python format pyds._format_type (set via datasets' own
  set_format): nothing, "numpy", "torch", "pandas", "arrow",
  "polars", …
• the julia jltransform field: py2jl, identity, or a user function.

The public API couples them into a single "format" slot (src/dataset.jl,
set_format!):

• "julia" ≡ (_format_type = nothing, jltransform = py2jl)
• any python format X ≡ (_format_type = X, jltransform = identity)

So the two slots are mutually exclusive through the API: setting "julia"
resets the python format to nothing; setting a python format installs
identity. The one combination that cannot be expressed is "python format X
and py2jl on top."

What prompted this

With datasets >= 4, ds["col"] returns a lazy datasets.Column, which we now
wrap in a lazy HuggingFaceDatasets.Column <: AbstractVector (added alongside
the datasets >= 4 support). It was observed that the "julia" format calls reset_format(),
so ds.pyds["col"] is always a Column.

Worth recording: resetting to nothing is not what forces the Column. In
datasets 4, Dataset.__getitem__(str) returns a Column for every format
except arrow/pandas/polars:

python format	ds["col"] returns
nothing (our "julia")	datasets.Column (lazy)
"numpy"	datasets.Column (lazy)
"torch"	datasets.Column (lazy)
"pandas"	pandas.Series (native)
"arrow"	pyarrow.ChunkedArray (native)
"polars"	polars.Series (native)

So decoupling the two formats would not avoid Columns in general — the lazy
Column is simply the v4 default for column access.

Points to reconsider

1. One format slot vs. two independent axes. Conceptually the python format
   and the julia conversion are orthogonal (what datasets produces vs. a
   julia-side transform). Exposing both is more faithful but roughly doubles the
   state space; most combinations are redundant (py2jl already absorbs numpy
   copylessly via DLPack) or ill-defined (py2jl has no branch for a pandas
   Series / arrow table / torch tensor). Current stance: keep the single slot;
   with_jltransform / MLUtils.mapobs are the intended composition points.

2. The one real limitation — numeric throughput. Under nothing, a numeric
   batch comes back as a python list and py2jl copies it element-by-element;
   under "numpy" it would be an np.ndarray → numpy2jl (zero-copy). This is
   the single case where "numpy" + py2jl genuinely wins.

3. Possible internal optimization (benchmark first). Rather than exposing a
   second format knob, consider whether the "julia" format should sit on
   "numpy" instead of nothing, so numeric columns transfer copyless. Tradeoff:
   image columns currently arrive as PIL.Image (→ Gray/RGB ImageCore
   views) under nothing; under "numpy" they'd be raw transposed arrays and we
   lose that clean path. Best validated with the perf/ harness on both numeric
   and image datasets before changing anything.

Non-goals / decision so far

• Keep the current format behavior as-is for now (intentional, documented in the
  guide's "The "julia" format and transforms" section).
• No API change until the numeric zero-copy question (2/3) is benchmarked and we
  decide whether the win justifies either an internal base-format change or a
  second, opt-in axis.