Electron migration#2
Open
kartikmandar wants to merge 76 commits into
Open
Conversation
- React frontend with MUI components (Header, Sidebar, Footer, RightToolbar) - 18 QuickLook pages, Pulsar, Modeling, Simulator, Utilities sections - FastAPI backend with Stingray service layer - Integrated log console, system resources, notifications - Python backend lifecycle management via Electron IPC - Pixi environment and dev scripts
- Add complete EventList load/save/delete/info API endpoints - Extend DataService with URL loading, file size checks, memory cleanup - Add NotificationToast component for global alerts - Update DataIngestion page with advanced options and event list management - Add sample data documentation (DATA_README.txt) - Update gitignore for large NICER dataset
- Add GTI validation checking for empty, invalid, and out-of-bounds events - Fix temp file cleanup in URL loading with try/finally - Fix FITS writing to use Astropy Table for OGIP compliance - Fix directory creation edge case for paths without directories - Display GTI warnings in UI with tooltip
- Move resource monitor, backend status, about dialog to RightToolbar - Merge help/contact/support into single Help menu - Add backend status dot indicator on processing icon - Remove Settings button (already in Header) - Delete Footer.tsx - no longer needed - Fix RightToolbar height to account for LogPanel
- Fix CPU percent always showing ~0 by adding interval=0.1 to psutil - Filter /api/status from uvicorn access logs to prevent log flooding - Increase polling frequency from 5s to 2s for faster updates - Remove non-functional CPU/RAM limit sliders from resources popover - Add Electron process resource monitoring via IPC - Enhance data service with better GTI validation
- Add /load-batch-stream SSE endpoint for streaming batch load results - Use asyncio.wrap_future + asyncio.wait for non-blocking async iteration - Files now appear in UI immediately as each one completes loading - Add console logging for SSE events, GTI warnings, and Stingray warnings - Capture Stingray library warnings during file loading - Update progress bar to show determinate progress (X of Y files) - Normalize CPU percentages by core count in resource monitoring
- Add LogStreamManager to capture Python logging and warnings - Add /api/logs/stream SSE endpoint for real-time log delivery - Add LogStreamClient in frontend with auto-reconnect logic - Connect LogPanel to SSE stream when backend is ready - Include /api/logs/test endpoint for development testing
- Add localStorage persistence for last loaded files with restore button - Browse now appends files instead of replacing selection - Add "Clear All" button for file selection - Selection clears after load attempt (success or failure) - Add GTI validation and inverted GTI fix in backend - Add comprehensive data quality validation checks - Add save format dialog and event notes support
- Add archive service for querying HEASARC data archive - Add archive routes and API client for archive operations - Add FileBrowserDialog component for local file browsing - Add HeasarcBrowserPanel for browsing remote archive data - Update data service with additional validation and features - Add pyvo dependency for Virtual Observatory access - Update pixi.lock with new dependencies
Wrap synchronous service calls with asyncio.to_thread() in data routes to prevent blocking the FastAPI event loop during file operations. This allows resource monitoring and other async tasks to continue while files are being loaded, saved, or processed. Affected endpoints: - POST /api/data/load - POST /api/data/load-url - POST /api/data/save - POST /api/data/load-by-time-range - POST /api/data/load-by-event-count - POST /api/data/metadata
There was a problem hiding this comment.
Pull request overview
Migrates the project to an Electron + React (electron-vite) desktop app architecture backed by a local FastAPI Python service, adding a new UI shell, API clients, and backend endpoints.
Changes:
- Added Electron main/preload processes with IPC plumbing and a Python backend lifecycle manager.
- Added a new React app structure (router, layout, placeholder pages), Zustand stores, and frontend API modules.
- Added a FastAPI backend with state/services, log streaming (SSE), archive/download helpers, and dev/build scripts/config.
Reviewed changes
Copilot reviewed 94 out of 103 changed files in this pull request and generated 31 comments.
Show a summary per file
| File | Description |
|---|---|
| tsconfig.node.json | Adds TS config for node/build tooling references. |
| tsconfig.json | Adds strict TS config, path aliases, and project references for renderer/electron code. |
| test_astropy_roundtrip.py | Removes an ad-hoc Astropy roundtrip test script. |
| src/types/electron.d.ts | Adds renderer typings for the preload-exposed Electron API. |
| src/store/uiStore.ts | Introduces global UI state store (panels, processing state, notifications, resources). |
| src/store/logStore.ts | Adds centralized log store + filtering and external log injection helper. |
| src/pages/Utilities/StatisticalFunctions/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Utilities/MissionIO/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Utilities/Misc/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Utilities/IO/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Utilities/GTI/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Simulator/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/VariableEnergySpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/RmsEnergySpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/PowerSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/LightCurve/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/LagEnergySpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/ExcessVarianceSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/EventList/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/DynamicalPowerSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/DeadTimeCorrections/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/CrossSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/CrossCorrelation/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/CovarianceSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/Coherence/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/Bispectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/AvgPowerSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/AvgCrossSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/AvgCovarianceSpectrum/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/QuickLook/AutoCorrelation/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Pulsar/Phaseogram/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Pulsar/PhaseFolding/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Pulsar/PeriodSearch/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/NotFound/index.tsx | Adds a 404 page for unknown routes. |
| src/pages/Modeling/ModelBuilder/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Modeling/MLEFitting/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Modeling/MCMCFitting/index.tsx | Adds placeholder “coming soon” page. |
| src/pages/Home/index.tsx | Adds a new home page with quick-access navigation cards. |
| src/main.tsx | Adds React entrypoint mounting <App />. |
| src/index.css | Adds global styles (scrollbars, animations, plot container sizing, titlebar regions). |
| src/components/layout/MainLayout.tsx | Adds app shell layout (header/sidebar/right toolbar/log panel/notifications). |
| src/components/layout/Header.tsx | Adds header with backend status, search UI, and controls. |
| src/components/common/PageTemplate.tsx | Adds reusable placeholder/“coming soon” template. |
| src/components/common/NotificationToast.tsx | Adds toast notifications driven by UI store. |
| src/api/timingApi.ts | Adds frontend client functions for timing endpoints. |
| src/api/spectrumApi.ts | Adds frontend client functions for spectrum endpoints. |
| src/api/logApi.ts | Adds SSE log stream client that pushes backend logs into log store. |
| src/api/lightcurveApi.ts | Adds frontend client functions for lightcurve endpoints. |
| src/api/index.ts | Re-exports API modules/types from a central barrel. |
| src/api/exportApi.ts | Adds frontend client functions for export endpoints. |
| src/api/client.ts | Adds fetch-based API client with Electron-backed dynamic port discovery. |
| src/api/archiveApi.ts | Adds frontend client for HEASARC search + SSE download streaming. |
| src/App.tsx | Adds router + theme + backend readiness context and wiring. |
| scripts/start-backend.sh | Adds helper script to start backend via Pixi. |
| scripts/setup-python.sh | Adds helper script to create venv and install backend requirements. |
| scripts/package.sh | Adds packaging helper for bundling backend resources into dist. |
| scripts/dev.sh | Adds unified dev launcher (backend + electron-vite dev). |
| scripts/build.sh | Adds build helper for Electron app build. |
| python-backend/utils/performance_monitor.py | Adds backend performance monitoring utilities. |
| python-backend/utils/log_stream.py | Adds Python logging/warnings capture and SSE streaming support. |
| python-backend/utils/error_handler.py | Adds standardized error handling/formatting utilities. |
| python-backend/utils/init.py | Exposes backend utility classes. |
| python-backend/services/timing_service.py | Adds timing operations (bispectrum, power colors, lags, coherence). |
| python-backend/services/state_manager.py | Adds backend in-memory state manager with memory cleanup helpers. |
| python-backend/services/spectrum_service.py | Adds spectrum operations (power/cross/dynamical spectra, rebin, list/delete). |
| python-backend/services/lightcurve_service.py | Adds lightcurve operations (create/rebin/list/delete). |
| python-backend/services/export_service.py | Adds export operations (CSV/HDF5/FITS + bispectrum CSV). |
| python-backend/services/base_service.py | Adds shared service base class (result shaping + error handling). |
| python-backend/services/init.py | Exports service classes for backend use. |
| python-backend/routes/timing_routes.py | Adds FastAPI routes for timing endpoints. |
| python-backend/routes/spectrum_routes.py | Adds FastAPI routes for spectrum endpoints. |
| python-backend/routes/log_routes.py | Adds FastAPI SSE routes for backend log streaming. |
| python-backend/routes/lightcurve_routes.py | Adds FastAPI routes for lightcurve endpoints. |
| python-backend/routes/export_routes.py | Adds FastAPI routes for export endpoints. |
| python-backend/routes/archive_routes.py | Adds FastAPI routes for HEASARC search + download-to-disk streaming. |
| python-backend/routes/init.py | Exposes route modules. |
| python-backend/requirements.txt | Adds pip requirements for the new backend. |
| python-backend/main.py | Adds FastAPI app startup, CORS config, status/shutdown endpoints, and port selection. |
| pixi.toml | Adds Pixi project/tasks/dependencies for backend + dev workflow. |
| package.json | Updates Electron build/package scripts and frontend tooling dependencies. |
| index.html | Adds renderer HTML entry with CSP and app metadata. |
| files/data/DATA_README.txt | Adds documentation for sample data included in the repo. |
| electron/pythonManager.ts | Adds backend process management and health/port handling for Electron. |
| electron/preload.ts | Exposes a typed, constrained IPC API to the renderer via contextBridge. |
| electron/menu.ts | Adds native app menu with navigation hooks. |
| electron/main.ts | Adds Electron main process window creation + backend startup + log buffering. |
| electron/ipcHandlers.ts | Adds IPC handlers for dialogs, filesystem, python control, app info, resources, etc. |
| electron.vite.config.ts | Adds electron-vite build configuration for main/preload/renderer. |
| .gitignore | Updates ignores for Node/Electron/Python/Pixi build artifacts and large data files. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ndling - Add comprehensive job management infrastructure: - Job models with status, progress, and result tracking - JobManager service for background task execution - Job routes with SSE streaming for real-time progress - Frontend job store (Zustand) and useJobStream hook - JobStatusPanel component for UI display - jobApi for frontend-backend communication - Improve log streaming: - Clear log history on new Electron session to prevent stale log replay - Remove unnecessary SSE delay for faster log delivery - Simplify UI components: - Streamline DataIngestion page - Clean up Sidebar, MainLayout, and RightToolbar - Simplify dev.sh startup script
…and locate_data fixes - Add obs_data passthrough (prnb, ra/dec) for mission-specific directory lookup - Construct RXTE archive URLs using proposal number and AO cycle estimation - Fix locate_data to use coordinate-based queries and proper API workflow - Add date/time column sorting and MJD-to-human-readable conversion in browser - Improve exposure column detection for NuSTAR (exposure_a) and XMM (duration) - Add debug test scripts for HEASARC column, locate, and URL testing
Pass port as positional arg instead of --port flag to match main.py's sys.argv[1] parsing. Scope pkill pattern to project-specific path to avoid killing unrelated python processes.
…rowser Add 5 new X-ray missions with catalog entries, FTP directory URL patterns, browse URLs, and mission-specific exposure handling (IXPE per-detector-unit, ASCA sequence_number obsid column).
- Fix broken _locate_observation_directory fallback that called query_mission_list with invalid params (always crashed with TypeError), replaced with ADQL query via query_tap - Remove dead query_mission_list call in get_observation_download_urls - Robustify file classification by stripping .gz/.bz2 compression extensions before pattern matching - Add search_by_obsid using ADQL TAP query (no coordinates needed) - Add post-query min_exposure and time_range filters to name/coordinate searches - Extract NICER processing_status and num_fpm from catalog results - Add ObsID search mode toggle, filter panel (min exposure, date range), and configurable max results to frontend - Show NICER processing_status as colored chip with tooltip explanation - Move resolved coordinates message to notification center
…nd Mode tooltips Only show FPMA instrument label on exposure chip when FPMB data is available (ADQL/ObsID search). For name/coordinate searches where query_region() doesn't return exposure_b, show plain exposure to avoid misleading users. Add explanatory tooltips to Mode column chips (SCIENCE vs SLEW). Include NuSTAR-specific UI enhancements: issue flag warnings, SLEW row dimming, and observation mode column.
…browser Extract per-instrument exposures/modes (PN, MOS1, MOS2) for XMM-Newton and detector/grating info for Chandra. Display status chips, instrument chips with tooltips, and conditional row dimming for unavailable data. Add Chandra auxiliary file patterns for archive file classification.
Redesign the entire frontend with an Observatory Dark aesthetic: - Deep space-inspired dark theme with teal-green (#00d4aa) and blue (#3b82f6) accents - JetBrains Mono for display/headings, IBM Plex Sans for body text - Glassmorphism cards, dialogs, menus with backdrop-filter effects - Grain texture overlay, glow hover effects, gradient borders - Staggered reveal animations and CSS starfield on Home hero - Refined light mode with matching design language - Stingray green retained as tertiary accent color - Terminal-dark LogPanel, mission-control Sidebar, glass Header - Remove all test and debug files from tests/ directory
…S window close - waitForReady() now polls up to 180s (was 30s) so cold imports of stingray/numba/astropy no longer trigger a spurious startup failure; fails fast if the spawned process exits and logs progress every 15s - on macOS, window-all-closed no longer tears down the Python backend; dock reopen re-sends python:ready or re-initializes, fixing the window stuck on "Starting..." after close/reopen
Add _power_to_lists/_finite_list helpers and update all five response-dict
constructions (create_power_spectrum, create_averaged_power_spectrum,
create_cross_spectrum, create_averaged_cross_spectrum, rebin_spectrum) so
complex arrays become {power: magnitude[], power_phase: phase[]} and all
non-finite values map to null — making every endpoint strictly JSON-safe.
…ctrum - _finite_list returns values.tolist() directly when all values are finite (0.06ms guard vs 240ms per-element loop on 500k points); the None-mapping loop now only runs when NaN/Inf are actually present - drop redundant astype(float) in _power_to_lists (_finite_list coerces dtype) - sanitize dyn_ps rows with _finite_list in create_dynamical_power_spectrum (frac norm + zero-count segments can yield NaN/Inf) - coerce dps.time to float64 before tolist(): Stingray stores time as np.longdouble, whose tolist() yields non-JSON-serializable longdouble scalars (caught by the new dynamical PS serialization test) - add pinning tests for _finite_list None-mapping, cross-spectrum magnitude/phase correctness, and dynamical PS strict-JSON serializability
Copy CrossSpectrum page with five deltas: component renamed to AvgCrossSpectrumPage, segment-size state/validation added, handleRun calls createAveragedCrossSpectrum, PageTemplate updated, and segment_size chip added to result header.
Add _overlap_error() helper to spectrum_service and timing_service. Guard create_cross_spectrum, create_averaged_cross_spectrum, calculate_time_lags, and calculate_coherence immediately after both event lists are fetched. Replaces cryptic "zero-size array to reduction operation maximum" ValueError with a human-readable message naming the two time ranges. Tests added for both service paths (TDD).
Implements the Heil et al. (2015) power-color analysis: a TDD'd helper computePowerColorRatios (PC1 = C/A, PC2 = B/D, null-safe) plus the QuickLook page, route, and sidebar nav entry.
Electron 28->41 (bumped in 5025a97) resolves unsandboxed preload module type Node-style: a CommonJS preload named .js under the root "type": "module" is parsed as ESM and crashes on require before contextBridge.exposeInMainWorld runs, leaving window.electronAPI undefined (file dialogs dead, silent port-fallback elsewhere). Verified with an offscreen Electron probe: preload-error before, electronAPI object after.
Plotly gl traces request contexts with failIfMajorPerformanceCaveat, so a renderer degraded to software compositing refuses them and plotly renders 'WebGL is not supported by your browser' into the plot div. Probe once per session with the same flag and transparently rewrite scattergl->scatter so every analysis page still draws (decimation caps keep SVG tractable). Fresh-instance probes confirm full ANGLE Metal GPU on this machine; the degraded state was instance-local and left no crash report.
regl (plotly's gl backend) generates draw commands with new Function; script-src 'self' made that throw EvalError, which plotly swallows and reports as 'WebGL is not supported by your browser' on every gl plot, despite healthy ANGLE Metal contexts. The QuickLook pages are the app's first gl traces, so the four-year-old CSP was never exercised. Verified by in-app probe: eval blocked -> message; eval allowed -> clean render.
.js-plotly-plot { height: 100% !important } overrode PlotlyChart's inline
height; inside auto-height cards the container collapsed to zero while the
plot still drew into it, so charts appeared blank until a window resize
forced a re-measure, and theme toggles re-collapsed them. Rule predates
any real chart in the app. Verified by in-page probe: plot div 0px tall
before, 440px from first paint after, GL pixels stable across theme-style
updates.
titleBarStyle hiddenInset draws the close/min/max buttons over the web content; the toolbar now reserves 76px on darwin (platform via IPC) so the logo no longer sits under them. Also wires the dormant titlebar-drag-region CSS so the frameless window is draggable by its header, with no-drag opt-outs on every interactive control.
The previous sx padding lost the cascade to MuiToolbar-gutters' media-query rule (equal specificity, later insertion), measured at 24px effective. A real flex spacer is immune to the cascade; first toolbar control now measures at x=68, clear of the inset buttons.
getWindowButtonPosition() can't report the OS default, so pin the lights explicitly at (16, 24) - vertically centered in the 64px header - and size the spacer against known geometry: cluster ends ~68px, spacer 72px puts the hamburger at x=84 with a 16px gap (was 68, where the hover ripple still grazed the green light).
Fixed pixel spacers drift against the OS-drawn traffic lights under page zoom (CSS px scale, the lights don't). Enable titleBarOverlay so the renderer can size the header inset from env(titlebar-area-x), which the compositor recomputes on zoom and fullscreen. Probe-verified at zoom 1.0/1.5/0.67: button stays 111-134 device px, always clear of the lights ending at ~68.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.