Add pinned polymer diffraction peak inference - #267
Draft
NJMarchese wants to merge 36 commits into
Draft
Conversation
Wire the existing quantem calibration tools into preprocess() (previously a
stub): image-center finding, ellipticity fitting, descan (CoM plane fit), and
detector-rotation estimation. Follows the class's lazy design -- parameters are
measured and cached (image_centers, ellipse_params, descan_origin,
detector_rotation_deg, sampling_inv_A), then consumed downstream by the polar
transforms rather than re-warping the raw 4D data.
- Ellipticity via fit_probe_ellipse on the mean DP -> metadata["ellipticity"]
- Descan + rotation via CenterOfMassOriginModel (calculate_origin,
fit_origin_background, estimate_detector_rotation)
- image_centers via find_central_beams_4d ("descent"/"grid"/"peaks") or the
CoM/descan field, selectable with center_source
- Register the new calibration attributes in __init__
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Render a boustrophedon cursor walk over the scan to an animated GIF: each frame pairs the real-space intensity map (cursor crosshair) with that position's diffraction pattern and detected Bragg-peak overlay, reusing the save_peak_figures rendering primitives. PIL-based writer.
…spacing axis - estimate_peak_windows(mode='intensity'|'count', log_scale=False): detect peak windows on the peak-count radial profile as well as intensity, and optionally on log1p(profile) so small peaks are not dominated by large ones. peak_info now carries 'mode', 'log_scale', and 'profile'. - peak_radial_intensity_plot / peak_radial_count_plot: add log_scale (log y-axis with a positive fill baseline) and show_d_spacing (top axis in real-space d-spacing (Å) = 1/q) when plotting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- estimate_peak_windows: printed per-peak summary now reports d-spacing (Å) = 1/q alongside the q values (center and window). - plot_peak_count_map: panel titles now include the d-spacing range (Å) under the q range (1/Å). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reorder BraggPeaksPolymer.preprocess() so calibration runs descan/rotation -> centering -> ellipticity (was ellipse first). Fitting the ellipse on the raw mean DP smeared the diffraction ring by the descan drift and biased the fit toward the central beam. The ellipse is now fit on a centered mean DP built by the new _centered_dp_mean() helper: when a fitted CoM model is available it reuses shift_origin_to()'s sub-pixel grid-sampler to align every pattern to the detector center on-device, otherwise it falls back to translating the plain mean DP by the average center offset. Centering (find_central_beams_4d) now runs without ellipse_params since the ellipse is not yet known. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace fit_probe_ellipse (which Otsu-thresholds and fits the brightest blob = the central beam, so it measured probe shape and latched onto a smeared/off-center beam) with a diffraction-RING fit using Karen Ehrhardt's angular-uniformity criterion. New _fit_ellipse_from_ring() holds the found center fixed and searches (b/a, theta) to minimise the azimuthal variance of a ring annulus in the polar transform (quantem.diffraction.polar_transform). It samples out at the ring radius and never touches the central beam, so a doubled / off-center / drift-smeared central beam no longer biases the ellipse. Details: - ring band auto-detected from the circular radial profile (skip central beam to first trough, take strongest ring beyond), or set explicitly via new ellipse_radial_min / ellipse_radial_max preprocess params. - coarse (b/a, theta) grid + local refine; output canonicalised to a/b >= 1. - centered mean DP now cached on self.dp_mean_centered (was ephemeral); preprocess caches self.ellipse_ring_band; show=True plots the DP plus before/after polar so the ring flattening is visible. - ellipse_threshold kept but unused (back-compat). Validated against polar_transform on synthetic elliptical rings (a/b and theta recovered to <0.5%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_centered_dp_mean used integer torch.round rolls. The per-pattern rounding residual (up to 0.5 px) does NOT average out when the origin spread is narrow -- every pattern rounds to the same integer, so the fractional part becomes a constant bias and the mean beam lands off the detector center by a fraction of a pixel (visible as the central beam sitting up/right of the geometric-center crosshair, and it also throws off the fixed-center ring ellipse fit). Replace integer rolls with bilinear splatting: each pattern's continuous shift (center - fitted origin) is split into floor + fraction, and its intensity is distributed over the 4 integer-shift corners with bilinear weights, grouped by floor shift for efficiency. Same low memory (one (Qy,Qx) accumulator + one batch), now centers to <0.01 px. Also fix the no-CoM fallback to average image_centers over valid (non-zero) positions only -- it is 0 outside the scan mask, which otherwise biases the fallback center toward the origin. Validated on synthetic beams: bilinear hits the detector center exactly at every origin spread; integer rolls drift up to ~0.4 px at narrow spread. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The centered mean DP still showed the central beam a few px off the detector center. Root cause: _centered_dp_mean shifted each pattern by the descan CoM origin (origin_fitted). The CoM is the centroid of the WHOLE pattern, so ring/background asymmetry (e.g. a bright corner) pulls it several px off the actual beam -- centering by it puts the CoM at the center and leaves the beam off. Center instead by image_centers, the angular-uniformity BEAM center (found by ring symmetry, immune to intensity asymmetry, and the center everything downstream already uses). Only ROI (in-mask, non-zero) patterns are averaged. Still sub-pixel bilinear, same low memory. Validated on a synthetic pattern with an asymmetric background pulling the CoM to (130,132): centering by CoM leaves the beam at (114.5,114.3) (~7 px off); centering by image_centers lands it at (107.9,107.9) on the (107.5,107.5) detector center. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The descan residual (origin_com_measured - descan_origin) showed a uniform ~1-2 px offset inside the ROI. Cause: CenterOfMassOriginModel fits fit_origin_background over the WHOLE scan, so out-of-ROI patterns (vacuum/substrate, meaningless CoM) drag the global plane and leave a constant offset inside the mask; hot/dead pixels contaminate it too. CenterOfMassOriginModel has no mask support, so preprocess now fits the plane itself over ROI patterns via new _fit_origin_roi (ordinary LS z = a + b*row + c*col per component, with sigma-clip passes to reject outlier CoM), then assigns com_model.origin_fitted and zeroes the residual outside the ROI (measured := fitted there) so estimate_detector_rotation's curl isn't contaminated by junk patterns either. Falls back to the full scan when scan_mask is None. Other centering steps already respect the ROI: find_central_beams_4d takes scan_mask, and the ellipse fit runs on the image_centers-centered (ROI-only) mean DP. Validated: with out-of-ROI junk + hot-pixel outliers, the global fit leaves a -13 px uniform ROI residual; the ROI fit brings it to -0.005 px and recovers the true plane exactly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_cartesian_peaks / save_polar_peaks / save_polar_data / save_peak_intensities did np.save(vector). A canon Vector is array-like, so numpy flattened it into an (Ry,Rx) object array of cells, losing the Vector's fields/units. On load, .item() only unwraps a size-1 array, so you got back a size-(Ry*Rx) object array instead of a Vector, and polar_transform_peaks raised "can only convert an array of size 1 to a Python scalar". Add _save_object() which wraps the object in a 0-d object array so np.save pickles the whole Vector; the existing size-1/.item() unwrap in the load_* methods then restores it. Route all four save_* through it. Note: files saved by the old code are already flattened and can't be reconstructed (metadata lost) -- re-run find_peaks_model and re-save. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
calculate_orientation_correlation sized its angular-mode batches from torch.cuda.mem_get_info(), which reports memory free on the *device* and is blind to torch.cuda.set_per_process_memory_fraction. Under a cap it budgeted headroom the process was not allowed to allocate and then died in fft2 with the card still mostly free: a 23.74 GiB cap against 60.65 GiB device-free raised OutOfMemoryError on a 100x100 scan at orientation_upsample=8. Headroom is now min(device_free, cap - memory_allocated). It is measured against *allocated*, not *reserved*: cached blocks no tensor is using are reusable, and after a failed run they can occupy most of the cap, which would otherwise report a zero budget and refuse to run. Also reserve room for theta_spectrum, which is allocated after the estimate and was previously unaccounted, and retry once after empty_cache() before giving up, so a failed run no longer poisons the next attempt. The MemoryError now reports the cap, live usage, spectrum reservation and device-free rather than a bare "0.00 GiB". Tests: tests/diffraction/test_orientation_correlation.py, 10 passed. Numerics unchanged, max relative difference 8.96e-08. Both failure modes reproduced on GPU and confirmed fixed: a cap saturated with stale cache now completes at 2.74 GiB under a 7.60 GiB cap, and a 1.14 GiB cap degrades to 182 small batches instead of failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the central-beam/probe-blob ellipse fit with a ridge fit on the diffuse ring. It measures the ring radius around every azimuth, jointly refines the residual center and the ellipse, and accepts the correction only when held-out angular sectors improve over a plain circle. Because it never touches the central beam, descan-drift smearing and doubling of the beam no longer bias the ellipse. Adds _fit_ellipse_from_ridge with polar_at / extract_ridge helpers and ellipse and circle residual models, wires it into preprocess() via ellipse_fit_method, and records the outcome in ellipse_fit_diagnostics (method, accepted, selected) alongside ellipse_ring_band. Tests: tests/diffraction/test_ellipse_ring_fit.py plus tests/diffraction/test_origin_finding.py, 21 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The correlation-equals-one boundary saturates: on real data (pg3T2 07-03-2024 scan 61) ring pair (2,2) climbs from 33.8 degrees at zero separation to 71.6 by mid-lobe, then flattens to 78.9. Fitting an unweighted straight line over the whole lobe let that flat tail dominate, which displaced the intercept off the measured boundary and biased the slope low. The fitted intercept landed up to 6.99 degrees away, visibly inside the blue region rather than on the gray baseline. Weight the fit exponentially towards short distances so the reported slope is the near-origin tangent, which is the physically meaningful quantity. Intercept error against the measured boundary at zero separation, across all six ring pairs: before +0.66 +3.14 -6.99 +4.70 -3.66 +6.89 (max 6.99) after -1.11 +1.40 -1.69 -1.09 +1.31 -1.39 (max 1.69) The 1/e decay defaults to SLOPE_WEIGHT_FRACTION (0.10) times the fitted distance span, chosen by sweeping the fraction against this dataset. slope_weight_scale=numpy.inf restores the previous unweighted fit exactly. R-squared is now weighted with the same weights and is not comparable to the old value. Adds slope_fit_intercept_degrees, slope_fit_effective_point_count (Kish) and slope_weight_scale to the metrics so the fit can be checked numerically. The fit line is drawn only out to three decay lengths, past which the boundary has saturated away from the tangent. Note this changes reported slopes, which were biased low: pair (2,2) 0.231 -> 0.371 deg/px (+61%), (1,1) +49%, (1,2) +46%, (0,2) -20%. Previously exported slope values need regenerating. Tests: tests/diffraction/test_orientation_correlation.py, 10 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The polymer ice flagger only tested "bright and in the q band, on one
six-fold lattice". Three additions, each off by default:
Sharpness gate. Ice reflections are small and sharp; polymer signal is a
larger dot or a broad diffuse region. Peaks carry no width, so measure it
from the polar volume: radial/annular FWHM above a local baseline, at each
candidate's (r, theta). max_width_r_invA / max_width_theta_deg gate the q
band before the lattice search, so broad peaks neither get flagged nor drag
the phi estimate. sharpness_mode picks intersection ("both", compact dots)
or union ("either", which also keeps thin streaks). sharpness_mask is public
so a tuning preview applies the same rule instead of reimplementing it, and
collect_peak_widths / measure_ice_peak_widths report the widths to tune from.
Multiple crystallites. A pattern can hold several crystallites at unrelated
orientations; one pass only ever saw the strongest. The matcher now peels:
claim the best-supported lattice, remove its peaks, look again, up to
max_crystallites. min_phi_separation_deg stops a single lattice being
re-found as a near-duplicate. IceFlaggerDebug.phi_deg lists what was found.
Folded theta. process_polar(two_fold_symmetry=True) folds theta to [0,180),
collapsing every Friedel pair onto one angle -- so a pair scored one bin and
was rejected by min_matches=2, and arms 3-5 were unreachable, making
min_matches>3 silently impossible. The matcher is now period-aware
(theta_period_deg, read off the BraggPeaksPolymer), and detect_ice raises on
an unsatisfiable min_matches rather than matching nothing. min_peaks_per_arm
recovers the distinction folding destroys: 2 demands a Friedel pair on an
arm and rejects a lone peak.
Also: plot_q_intensity_density shades the candidate region as a rectangle
instead of drawing bare window edges.
Tests: 77 pass in tests/diffraction (28 new in test_polymer_ice.py).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Folding theta to [0,180) for two-fold symmetry discarded which half of the circle a peak came from, so a Friedel pair became indistinguishable from two peaks that merely sit close together. Peaks are small, so keep the full angle: polar_transform_peaks now emits a "theta_unfolded" column alongside the folded "theta". Nothing else reads it by position -- lookups go through fields.index() -- so the extra column is transparent. IceFlaggerParams.require_friedel_pair uses it: an arm counts only if it holds two peaks whose unfolded angles differ by 180 +/- dtheta_deg. That is the test min_peaks_per_arm can only approximate, since a count of two is also satisfied by two neighbours within dtheta_deg. The field is optional -- detect_ice raises a clear "re-run polar_transform_peaks" error only when the strict test is requested on a vector that predates it. Tests: 80 pass in tests/diffraction. test_bragg_peak_polar_transform_inverts_ ellipse_mapping updated for the new column, and now pins the field order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to the sharpness gate, from working it against a real polymer scan. Annular width is the discriminator; drop sharpness_mode. Ice is annularly sharp both as compact dots and as radial streaks (narrow in theta, extended in q), while polymer at the same q is an annularly broad arc. The radial axis does not separate ice from polymer -- dots and arcs measure alike there -- it only separates streaks from dots, so a radial ceiling costs the streaks and buys nothing. That makes the "both"/"either" combining rule pointless: with one ceiling the modes are identical, and "either" silently defeated a tight annular ceiling for any radially sharp peak. Ceilings are now simply ANDed. Width measurement is noise-robust. The half-maximum walk stopped at the first sample below half, so one downward fluctuation on a weak diffuse arc ended it early: a true 40 degree arc measured 10-30 degrees at realistic SNR, i.e. as narrow as ice. The profile is now Gaussian-smoothed (sigma 1 bin) and the crossing must persist for 3 samples. The kernel is removed in quadrature -- exact for a Gaussian convolved with a Gaussian, which is why it is a Gaussian and not a boxcar; a boxcar leaves a residual bias on sharp peaks. Median over 8 seeds, true 40 degrees at SNR 2: 10.5 -> 43.6, while a true 5 degree peak stays at 6.0. Also widened the annular window to 90 degrees and dropped the baseline quantile to 0.05, since a window the feature fills puts the baseline partway up it and saturates the measurement near 39 degrees. Parameters validate on construction. Every confusion this cost came back as "nothing was flagged" rather than an error. Module docstring now states the four criteria and the folding constraints. Tests: 89 pass in tests/diffraction. The synthetic polar fixture sampled q at 0.025 1/A per bin, too coarse for the default radial window; now 0.005. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ce flagger Choosing IceFlaggerParams against a real dataset needs plots, not just the algorithm: a selection box over the sharpness ceilings, ice-band orientation histograms for all/kept/removed peaks, a widget view of either side of the split, and a per-peak FWHM probe that shows the cuts each width is measured from. These started as notebook cells, which meant every scan directory carried its own drifting copy; in the package any notebook can import them. Free functions taking (bp, params) explicitly rather than methods on BraggPeaksPolymer, which is already 79 methods. The algorithms stay in polymer_ice; this is only the interactive layer. Nothing mutates bp -- IcePeakView delegates to it so the widget can show a peak subset without the caller's object being altered. The quantem.widget import is deferred into the one function that needs it, so the module imports without that package. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
None of this was polymer-specific. Measured inside exact AST method boundaries, the calibration cluster referenced `self` only to reach a Dataset4dstem, a peak Vector, a device string, and a slot to deposit diagnostics into -- all general core types. The two ellipse fitters take a plain 2D array and a centre, so they apply to any diffuse or crystalline ring from any instrument; `_fit_origin_roi` did not reference `self` at all. New `ellipse_fitting.py` holds `fit_ellipse_from_ring` (angular variance) and `fit_ellipse_from_ridge` (joint centre + ellipse), returning an `EllipseFit` dataclass. Returning the fit instead of assigning `self.ellipse_fit_diagnostics` was what actually completed the extraction: that assignment was the only thing tying 837 lines of arithmetic to the class. `EllipseFit.diagnostics` rebuilds the flat mapping the class has always published, so `bp.ellipse_fit_diagnostics` keeps its exact key set -- pinned by new tests for both methods. Origin/descan work moves to `polar_transform.py`, which already owned three origin finders, rather than to a third location: `find_central_beams_4d`, `find_central_beams_from_peaks_4d`, `fit_origin_roi`, `centered_dp_mean`. The dataset, peaks and device are now parameters. `find_central_beams_4d` stays on the class as a forwarding wrapper since it is public API; every internal caller already used keyword arguments. BraggPeaksPolymer: 79 -> 74 methods, 6798 -> 5691 lines. bragg_peaks.py drops below 8k. test_ellipse_ring_fit.py no longer constructs the class via object.__new__ to reach the fitters; the one remaining use there legitimately exercises preprocess argument validation. Tests: 488 passed, 10 skipped across the full suite (102 in tests/diffraction). Verified separately that both fitters still recover a synthetic ellipse (true ratio 1.111 -> 1.105 ring / 1.117 ridge; true theta 35 deg -> 36.0 / 35.3). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second extraction, same pattern as the calibration one. New `peak_visualization.py` holds the seven plotting functions whose only tie to the class was reading one or two attributes: the radial intensity and count profiles, the per-position count and histogram maps, the interactive pattern browser, figure export, and the pattern grid. They take the arrays and peak vectors they draw, so they work for any 4D-STEM dataset. All seven are public API and three appear in 51 notebooks, so each keeps a forwarding method on the class. The wrappers take *args/**kwargs and prepend the data, which preserves positional calls -- verified that the notebooks' `bp.plot_peak_count_map(peak_windows, return_values=True)` still lands `peak_windows` on `q_ranges`. Five shared module-level helpers moved too. The three display helpers (`_normalized_dp`, `_resolve_intensity_map`, `_intensity_display_limits`, plus `_mean_intensity_map`) live with the plots and are imported back for the three peak-overlay methods that stay. The two ragged-Vector field accessors went to a new `vector_fields.py` instead: they are needed by `estimate_peak_windows` and `make_orientation_histogram`, which are not visualization, and importing them from a plotting module would have been the wrong direction -- or a cycle. Found and fixed a bug the extraction introduced: threading `self.peak_coordinates_cartesian` in as `peaks` turned `peaks = self.peak_coordinates_cartesian[i, j].array` into `peaks = peaks[i, j] .array`, shadowing the parameter and failing on the second scan position. Only two of the seven functions had any test coverage -- including none of the three used by 51 notebooks -- so this was invisible. Added `test_peak_visualization.py` covering the five untested ones through both entry points, which is what caught it, and AST-scanned the rest for the same shadowing pattern (no others). Also repaired `test_bragg_plotting_and_save_smoke`, which monkeypatched `interactive_output` on `bragg_peaks` only; `plot_interactive_image_map` now resolves it from `peak_visualization`, so the patch had silently stopped applying. It now patches both modules. BraggPeaksPolymer: 5691 -> 4812 lines (79 -> 74 methods across both steps, since the public plotting entry points remain as wrappers). bragg_peaks.py 7103 -> 6162, down from 8206 at the start. Tests: 495 passed, 10 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written before the orientation extraction so it pins current behaviour rather than post-extraction behaviour. make_flowline_map and make_flowline_rainbow_image appear in 51 notebooks each and had no tests; make_flowline_combined_image, make_flowline_rainbow_legend and the two intensity helpers had none either -- 678 lines with no coverage, which is what let a shadowing bug through in the previous extraction. Covers the pipeline end to end on a small synthetic orientation histogram with a coherent stripe (a uniform histogram seeds no flowlines), plus a set/get round-trip for the interpolating intensity helpers. Note the flowline map is asserted to be *concentrated* in the seeded band rather than exactly zero outside it: the tracer steps along directions and set_intensity spreads weight across neighbouring bins, so a little legitimately bleeds out. Tests: 112 passed in tests/diffraction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third extraction. The cleanest cluster yet: of the 1564 lines, 678 referenced `self` not once -- the three flowline renderers and both intensity helpers were methods purely by accident. Nothing in it is polymer-specific; orientation histograms and flowlines apply to any material with anisotropic scattering. New `orientation.py` holds the pipeline -- make_orientation_histogram (threaded polar_peaks + peak_intensities), make_flowline_map, and the rainbow/combined/ legend renderers -- plus `_get_intensity`/`_set_intensity`, now private module helpers rather than public methods, which removes make_flowline_map's only tie to the class. `plot_orientation_correlation` went to `orientation_correlation.py` beside `calculate_orientation_correlation`, taking SLOPE_WEIGHT_FRACTION with it. All eight public names keep forwarding methods. `plot_orientation_correlation`'s wrapper preserves its self-fallback, so `bp.plot_orientation_correlation()` with no arguments still plots the volume the object last computed. `calculate_orientation_correlation` stays put: it writes orient_corr/orient_corr_pairs onto the object, and already delegated its maths. Verified per the protocol in dp_peak_detection's HANDOFF: - Baseline tests written FIRST (e80ffa7) for the 678 previously uncovered lines. - AST scan for rebinding of injected parameter names: none. - refactor_diff_check.py: 23/23 bit-identical against baseline 1265551, the seven new orientation checks included. One bug caught by the baseline tests: the generated wrapper prefixed `_` onto the already-underscored `_set_intensity`, producing `__set_intensity`, which Python name-mangles inside a class body. Fixed. Exactly the class of failure the tests-first ordering is meant to catch, and it would have been invisible before e80ffa7 since these had no coverage. BraggPeaksPolymer: 4812 -> 3296 lines. bragg_peaks.py 6162 -> 4648, from 8206 at the start of the refactor. Tests: 500 passed, 10 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written before extracting them. Neither resize_images nor _postprocess_single had any tests, and between them they bracket the whole ML detection path: one prepares every pattern for the network, the other turns its two output channels into peak coordinates and intensities. test_resize_images_shape_and_intensity_convention pins the existing intensity convention rather than the one I expected. Bilinear interpolation to half the linear size already divides the sum by 4; the code then multiplies by scale_factor = (16*16)/(32*32) = 1/4 as well, so the total comes out 16x smaller. Count conservation would need the reciprocal (x4). Presumably this matches the convention the network was trained under -- pinned as-is, deliberately not changed, and called out in the docstring so the next reader does not assume the sum is preserved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourth extraction, and a deliberately small one -- the honest finding here is how little of this cluster should move. Of the 617-line ML cluster, only 115 lines are cleanly extractable: resize_images_for_model (prepares patterns for the network; read only self._final_shape) and peaks_from_model_output (turns the two output channels into coordinates and intensities; referenced self not at all). Both go to peak_detection.py beside the blob detection and refinement they feed into, so the module now covers the whole detection path rather than just its middle. The other nine stay, and should. find_peaks_model, infer_peaks_single, adapt_batchnorm, ensure_normalization_params, set_model_weights, _infer_train_batch_output, _invalidate_inference_caches, prepare_inference and resize_data read AND write instance state -- the model, the resolved normalization parameters, the BatchNorm-adaptation flag, the live chunk cache. That is a model lifecycle, which is what an object is for; free functions would mean threading a mutable context through all of them, which is worse than the class. save_peak_animation also stays: it calls infer_peaks_single and shares seven peak-overlay drawing helpers with plot_interactive_peak_map and save_peak_figures, so it belongs with that group's render-context decision. Verified per protocol: baseline tests first (b199e4d), AST scan for injected-parameter shadowing (none), refactor_diff_check.py 26/26 bit-identical. BraggPeaksPolymer: 3296 -> 3193 lines. bragg_peaks.py 4648 -> 4550. Tests: 504 passed, 10 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A survey of what was left in bragg_peaks.py turned up something the earlier cluster analysis had missed entirely: ScanMaskEditor is a *separate* 885-line class with 46 methods, sharing the file with BraggPeaksPolymer for no reason. Its only mention of BraggPeaksPolymer is a button tooltip string, and it uses none of the module's helpers. So this is a file split rather than a refactor -- no signature changes, no parameter threading, no shadowing risk. bragg_peaks.py now contains exactly one class. 3663 lines, from 8206 when the refactor started. All three import paths still resolve to the same object, verified: quantem.diffraction.ScanMaskEditor (what the tests use), quantem.diffraction.bragg_peaks.ScanMaskEditor (preserved by the import that edit_scan_mask needs anyway), and the new quantem.diffraction.scan_mask_editor.ScanMaskEditor. edit_scan_mask and create_interactive_circular_mask stay on the class: the editor is constructed as ScanMaskEditor(self), so those are factories bound to the analysis whose mask they edit, which is a legitimate reason to be a method. Tests: 504 passed, 10 skipped. ScanMaskEditor already had coverage (test_scan_mask_editor.py), so no new baseline tests were needed for this step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 16x shrink has no effect on results, and not merely by train/inference consistency -- the factor is annihilated. Resize precedes normalization everywhere, and every normalization strategy here starts with a per-image min-max, which is invariant under positive uniform scaling. Both reported intensity fields are post-normalization, so no pre-normalization absolute intensity reaches any output. Replaces the speculative note with the proof, and records the one genuinely latent concern: scale_factor depends on input detector size, so a future normalization with a fixed absolute divisor would make it matter, differently per dataset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written before extracting it. estimate_peak_windows appears in 51 notebooks and had no tests, and it decides the q-windows that every count map, radial profile and flowline family downstream is computed over -- a silent change would shift every result without failing anything. Covers recovery of three injected radial families, window bracketing and clipping plus the min_width floor, intensity/count mode agreement under uniform intensities, log compression, the empty-result shapes, and mode validation. Also pins the peak_info key set including the profile/intensity_profile alias, which is the same array under two names and easy to drop by accident when moving code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Appended to peak_detection.py, which now covers the detection path end to end: model input prep, blob detection and refinement, output post-processing, and the radial families the peaks cluster into. Two attribute reads threaded in as leading parameters, both documented in the existing numpydoc block; the class keeps a forwarding method since 51 notebooks call it. Verified per protocol: baseline tests first (90cf2cb) -- it had none, despite deciding the q-windows every count map, radial profile and flowline family downstream is computed over. Then AST scan for injected-parameter shadowing (none), then refactor_diff_check.py: 33/33 bit-identical, with seven new checks covering the returned centres, windows, and five peak_info arrays separately. BraggPeaksPolymer: 3193 -> 3050 lines. bragg_peaks.py 3663 -> 3521. Tests: 511 passed, 10 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tion QA grid The two interactive_probe_selector methods (275 lines) existed to build a list of (ry, rx) coordinates. That now lives in the viewer people actually use (quantem.widget 3f364b7), so they are deleted rather than moved. Verified unreferenced first: no callers in quantem, in any notebook, or in the widget repo. The two hits in dp_peak_detection are older vendored copies of the whole class, which carry their own definitions and are unaffected. visualize_peak_detection is extracted to peak_visualization.py -- but it had to be repaired first, because it did not run at all. It called len() on a Vector scan cell and indexed one with [:, 0]; both raise against the current Vector API, which needs .array. Any call failed on the first pattern. That is presumably why it was unused, and why nothing noticed: it had no tests. Repaired minimally (.array), which cannot regress behaviour since there was none, and it now has baseline coverage. Also dropped a per-pattern debug print and several commented-out alternate renderings that referenced instance attributes -- meaningless in a free function and non-executable either way. Its six attribute reads are threaded as leading parameters and documented in the numpydoc block; the class keeps a forwarding method. Draw-only, so nothing to compare numerically -- recorded in refactor_diff_check's DRAW_ONLY with a note that no pre-move behaviour existed. AST scan clean; the wider check stays 33/33. BraggPeaksPolymer: 3050 -> 2649 lines, 74 -> 72 methods. bragg_peaks.py 3521 -> 3121, from 8206 when the refactor started. Tests: 513 passed, 10 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The selection could only be axis-aligned, so a feature lying diagonally across the scan had to be bracketed by an oversized upright box. Adds an Angle control (degrees, 0-180 since 180 is the same selection as 0) plus an initial_angle argument threaded through BraggPeaksPolymer.edit_scan_mask. The mask rotates the sampling grid rather than the shape, so the axis-aligned inside-tests are untouched. The overlay passes the same angle to the Ellipse and Rectangle patches, with rotation_point="center" so the rectangle turns about the selection centre rather than the corner its xy names. A circle is rotation invariant, so the control is hidden for it and .angle reports 0. Angle is persisted: schema bumped to 3, and schema 1/2 files still load with angle 0, since they were axis-aligned by construction. The invariant worth testing is that the drawn outline agrees with what will actually be selected -- a rotation applied to the mask but not the overlay, or with the opposite sign, would silently mislead. Parametrised over three geometries and four angles, comparing the patch's own inside-test against the mask on every scan pixel: ellipse agrees ~100%, rectangle and square 97-98%, the residual being the mask's inclusive edges versus the patch drawn a half pixel outside. One of my own tests was wrong first: I asserted the 45 degree long axis lay on the anti-diagonal, and it lies on the main diagonal. Which screen diagonal that is depends on the display origin, so the assertion now records the actual behaviour and defers the real question to the outline-agreement test. Also updated test_controls_follow_probe_directions_and_apply_is_explicit, which pinned the image row by index; the angle row now sits above it, and the assertions name each shape-control row rather than just counting. Tests: 531 passed, 10 skipped (18 new in test_scan_mask_editor.py). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the lobe The 50% radial distance came back NaN for orthogonal ring pairs. The metric was pinned to a relative orientation of zero, but those pairs correlate at the other end of the axis, where their zero cut is a shallow anticorrelated plateau that need never reach the half level. Investigating that turned up a larger error. Orientations are pi-periodic, so num_theta bins cover 180 degrees and the retained num_theta // 2 + 1 lags are spaced 180 / num_theta apart: the relative-angle axis always ends at 90 degrees. It was labelled 0-180, so every annular distance and every slope was a factor of two too large. Radial distances were unaffected. Verified to 2e-16 at num_theta = 180 and 360 against a synthetic with a known angular offset, where the cross-correlation is provably the autocorrelation shifted by that offset. The lamellar/backbone lobe is at 90 degrees, perpendicular, as expected physically rather than the "180 degrees" previously displayed. Three changes: - MAX_RELATIVE_ANGLE_DEGREES = 90 drives the angle axis, the imshow extent and the slope-fit clipping. - Intercepts reference the pair's lobe. Candidates are the two ends only, parallel or perpendicular. An earlier revision took a free argmax; it selected interior noise on 56 of 181 real pairs and swung decay lengths 0.09x-6.2x, so interior maxima are deliberately not eligible. - An intercept outside the measured window is censored rather than silently NaN: *_censored flags and *_lower_bound values let a correlation that outlasts the scan be told apart from one that could not be measured. Across 18 cached volumes (181 pairs): 108 lobes parallel, 73 perpendicular, 0 interior, 0 silent NaNs, 25 censored and bounded. Tests: four added covering axis span against ground truth, lobe reference, interior-noise rejection, and censoring. Three existing tests are updated — their fixtures built synthetic boundaries on a 0-180 axis, so the corrected labelling halved their expected slopes; their local angle axes now use 0-90. Full suite 536 passed, 10 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RNGMixin moved from core/utils/utils.py to core/utils/rng.py in c1a8abc (2025-10-07); augment_dp was never updated, so importing DPAugmentor has been broken since. Path-only change: RNGMixin is byte-identical across the move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BraggPeaksPolymer, peak detection, angular-origin/polar transforms, and calibrated flowline/figure helpersdev/polymer overlap reconciliationThe default paper record remains intentionally unavailable for network download until its immutable public Zenodo record exists. This PR should remain draft until that URL is pinned.
Verification
pytest -p no:cacheprovider -q tests/diffraction tests/datastructures/test_dataset4dstem.py(32 passed)(1, 2, 256, 256)inference passedScope exclusions
Grain clustering, generator/training launchers, research notebooks, generated results, and the private Zenodo archive are not included.