Parallel-SpotOptim UX: real progress labels, TensorBoard tuning visualization, write_image-safe figures#195
Merged
Merged
Conversation
…l SpotOptim search With SpotOptim n_jobs != 1 every objective evaluation runs in a worker process holding a dill copy of the closure, so a label derived from len(all_params) was stuck at 'config 1/N' on every bar. A multiprocessing.Manager counter + lock (whose proxies survive the dill round-trip and reconnect from spawned workers) now provides the true cross-process evaluation count. The parent-side 'SpotOptim trials' bar is created in sequential mode only — in parallel mode its update() ran on worker dill-copies and the parent bar printed a stale 0% line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mStrategy Forward optional config attributes tensorboard_log / tensorboard_path / tensorboard_clean through kwargs_spotoptim into the SpotOptim constructor. Users set them as plain attributes on the config (the safe package's config classes carry no __slots__), so spotforecast2_safe stays untouched. Unset attributes are skipped so SpotOptim's own defaults remain in charge. Live per-eval scalars under parallel tuning (n_jobs_spotoptim != 1) require spotoptim >= 0.12.8, which logs infill evaluations parent-side; the real-run test skips on older versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kaleido v1 serialises figures via orjson, which rejects the raw pandas Timestamps that tz-aware indexes leave in trace x arrays (object dtype) and in shapes/annotations/axis ranges — any direct fig.write_image() call failed with 'Type is not JSON serializable: Timestamp'. Coerce all datetime values to ISO-8601 strings at figure-build time (date axes accept them natively; rendering unchanged). save_to_file keeps its JSON round-trip for traces added after make_plot(). A full pio.from_json(fig.to_json()) round-trip was rejected: plotly 6 re-encodes numeric arrays as base64 bdata dicts, breaking consumers that read trace.y. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0.12.8 logs infill evaluations parent-side in steady-state parallel runs; with the floor bump the tuning-visualization feature is live under n_jobs != 1 and the gated end-to-end test executes. Also add a commented uv.sources template for a local editable spotoptim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Collaborator
Author
|
🎉 This PR is included in version 5.1.0-rc.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Collaborator
Author
|
🎉 This PR is included in version 5.1.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four commits fixing/finishing the parallel-tuning experience:
config k/Nprogress labels were frozen atconfig 1/Nunder SpotOptimn_jobs != 1: every evaluation runs in a worker holding a dill copy of the closure, so parent-side list length never advances. Amultiprocessing.Managercounter+lock (proxies survive the dill round-trip and reconnect from spawned workers) provides the true cross-process count. The parent-sideSpotOptim trialsbar is sequential-only (its worker dill-copies printed stale bars).SpotOptimStrategyforwards optional config attrstensorboard_log/path/cleanthroughkwargs_spotoptiminto SpotOptim. Set as plain attributes (cfg.tensorboard_log = True) — sf2-safe untouched. Docs section added tospotoptim_intro.qmd.fig.write_image()no longer dies withType is not JSON serializable: Timestamp(kaleido v1/orjson). Apio.from_jsonround-trip was rejected: plotly 6 re-encodes arrays as base64 bdata, breakingtrace.yconsumers.spotoptim>=0.12.8(logs infill evals parent-side in parallel runs) so live TB works undern_jobs=-1; ungates the end-to-end test.Verification
uv run pytest tests/ -q: 1165 passed (incl. end-to-end: parallel run withtensorboard_log=Trueproduces event files against released spotoptim 0.12.8; progress-label integration test fails on pre-fix code)quarto render --no-cachegreen🤖 Generated with Claude Code