Lite gets the Query Store backfill off switch (#2167 parity)

CHANGELOG.md

@@ -14,7 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Every previously-hardcoded alert threshold is now a real setting** ([#2107], the split-out from gotqn's #2101 - "it was fine to hardcode these for development but any serious monitoring allows configuring of alert thresholds") - six new knobs ride the store control plane (V55), the Viewer's Settings window, and `get_alert_settings`/`update_alert_settings`, clamped on read like their siblings: the monitor store volume's self-alert warning percent (was 10), the Collection Stopped staleness window (was 30 minutes) and consecutive-failure fast path (was 10), the low-disk CRITICAL severity tier's percent and GB floors (were 3% / 2 GB - these grade the target-volume alert in BOTH apps, and Lite reads its pair from `settings.json` as `alert_disk_critical_free_percent` / `alert_disk_critical_free_gb`), and the analysis notification cooldown (was a hardcoded 360 in Darling while Lite always honored a configured value - the parity gap closed). MCP shape: `low_disk.critical_free_percent` / `low_disk.critical_free_gb`, a new `self_alerts` group, and `analysis.notify_cooldown_minutes`.
 - **Per-database collection timing now separates server think-time from row streaming** ([#2164]) - the per-database Query Store line reports `sql:Xms = wm:Wms + open:Yms + drain:Zms`, where wm is the watermark refresh (a monitor-store round trip the timer already started before), open is everything before the first row arrives, and drain is streaming those rows to the collector. This exists because of a measurement that overturned an assumption: cutting the text budget from 64 MB to 12 MB on a production server moved 5x fewer bytes and roughly 7x fewer rows, and the batch clock did not move at all. That says the cost lives upstream of shipping - in Query Store's own aggregation before the first row - which no client-side budget or payload trimming can shorten. The blended number could not show that, so the split is now visible: a pass that is nearly all `open` needs the server-side query narrowed, and a pass that is mostly `drain` is the one a smaller budget or a shorter network path helps.
 - **The two collector memory bounds are now operator knobs** ([#2164], [#2170]) - the per-database Query Store text budget (was a hardcoded 64 MB) and the fleet sweep width (was a hardcoded 4 servers) ride the store control plane (V59), the Viewer's Settings window, and the service's live reload, clamped [4,256] MB and [1,16] on read. Defaults reproduce the old constants exactly, so an upgrade changes nothing until a dial moves. Why both at once: peak transient memory is roughly the two multiplied, so an operator moving one needs the other in front of them. Lower the budget when the monitored fleet is a network hop away - the budget bounds memory, but it also sets how long one collector query holds the monitored server open draining to the client, which over a cross-region link is the tenant-visible cost (a smaller budget trades catch-up latency for shorter statements, never data, because every cut is resumable). Raise the sweep width when a large fleet queues behind 4-wide collection on a host with headroom, which is what makes the Fleet Health screen report staleness while every collector reports healthy. Narrowing the width never interrupts a running collection - the retiring permits are absorbed as bodies finish.
-- **The Query Store backfill has an off switch** ([#2167]) - the #2058 backfill previously ran unconditionally, and during a fleet consolidation a freshly restored database's imported catalog put it into sustained byte-budget drains against a cross-region production primary with no way to stop it short of disabling plan capture everywhere. `config_service.query_store_backfill_enabled` (V58, default on) is read live by the service's backfill loop - flip it in the Viewer's Settings window (new checkbox beside plan capture) and the loop idles from its next cycle, no restart; re-enabling resumes exactly where the watermarks left off. Live collection is never affected.
+- **The Query Store backfill has an off switch** ([#2167]) - the #2058 backfill previously ran unconditionally, and during a fleet consolidation a freshly restored database's imported catalog put it into sustained byte-budget drains against a cross-region production primary with no way to stop it short of disabling plan capture everywhere. `config_service.query_store_backfill_enabled` (V58, default on) is read live by the service's backfill loop - flip it in the Viewer's Settings window (new checkbox beside plan capture) and the loop idles from its next cycle, no restart; re-enabling resumes exactly where the watermarks left off. Live collection is never affected. Lite gets the same control as a Settings checkbox ("Fill Query Store history gaps in the background"), read live so it takes effect without restarting the app - the store column exists on the Darling side because a headless service has no window to click.
 - **The store measures its own background jobs** ([#2136], the visibility half) - the hourly #2068 self-metrics sweep now writes one row per TimescaleDB background job (object_kind `background_job`, V56 columns): last run duration, schedule interval, total runs, total failures. Why: the store's heaviest recurring work is its own job machinery - on the production 52-server store the four most expensive jobs are all the query_store_stats family (compression 157s, interval_hourly refresh 96s) - their runtimes scale SERIALLY with raw volume (the finalize hash-aggregate runs in one process), and a job that outgrows its own cadence compounds refresh lag silently. With the interval stored beside the duration, 'how close is each job to its ceiling' is one division over a 400-day series instead of archaeology - the number an onboarding wave moves first. A threshold alert on the series is the issue's next half.
 - **The store alerts when its own background jobs outgrow their schedule** ([#2136], the alert half) - a new fleet-level self-alert, Store Job Over Cadence: WARNING when a job's last successful run reaches a store-backed percent of its own schedule interval (V57 knob `store_job_cadence_warn_percent`, default 25, clamped 5-100, on the Settings window and `get_alert_settings`/`update_alert_settings` under `self_alerts`), CRITICAL fixed at 100 - past that the job is still running when its next run is due, so runs back up behind each other and everything it maintains (CAGG freshness, compression, retention) falls further behind every cycle. Judged hourly from timescaledb_information.job_stats on the same connection as the compression-health check, successful runs only (a failed run's duration is not a cadence signal); standing condition with cooldown re-fires and a Store Job Cadence Recovered resolution row; the alert text points at the V56 duration series in collect.store_metrics for the trend. Calibration: the production 52-server store's worst job runs at ~7% of cadence, so the default warns at 3.5x today's ceiling but far ahead of real compounding.
 - **The #2136 capacity model is now proven by a synthetic scale test, not asserted from one observation** - a live end-to-end drives a throwaway hypertable's compression job at 1x and then 4x row volume (parked policy, deterministic run_job - the #1888 discipline, so the scheduler can never race the measurement) and pins the whole loop: job runtime GROWS with volume (monotonicity, not a ratio - runner jitter owns the constant factor, the direction is the claim), the V56 telemetry series records both readings in order, and the Store Job Over Cadence alert fires its Critical tier from REAL store readings once the schedule interval is shrunk under the measured duration.

Lite.Tests/AnomalyDetectorTests.cs

@@ -31,6 +31,26 @@ public class AnomalyDetectorTests : IClassFixture<SharedDuckDbFixture>, IDisposa
     private static readonly DateTime _analysisEnd = _now;
     private static readonly DateTime _analysisStart = _now.AddHours(-4);
 
+    /* #2177: the start of a seeded baseline day, floored to the HOUR.
+
+       Every seed helper below writes several samples spanning ~21 minutes from this point. They used to
+       start at whatever time-of-day _analysisStart inherited from the wall clock, so when a CI run put
+       _analysisStart within 21 minutes of midnight the span crossed a date boundary and each intended
+       'day' contributed TWO distinct dates — doubling the distinct-day count the baseline-quality gate
+       counts, which flipped a deliberately-thin (2-day) baseline into a trustworthy one and sent the
+       detector down the z-path instead of the absolute fallback. Deterministic failure for runs between
+       03:39 and 04:00 UTC.
+
+       Flooring to the hour rather than midday-anchoring (#1972's discipline elsewhere) is deliberate:
+       the Full baseline tier buckets by hour AND day-of-week, so the seeds must keep _analysisStart's
+       hour and weekday to land in the same bucket the analysis window reads. Starting at :00 keeps both
+       while making a 21-minute span unable to leave the hour, let alone the date. */
+    private static DateTime SeedDayStart(int daysBack)
+    {
+        var day = _analysisStart.AddDays(-daysBack);
+        return day.Date.AddHours(day.Hour);
+    }
+
     private long _nextId = -1;
 
     public AnomalyDetectorTests(SharedDuckDbFixture fixture)
@@ -457,7 +477,7 @@ private async Task SeedBaselineCpu(int avgCpu, int variance)
         var rng = new Random(42);
         for (int day = 1; day <= 14; day++)
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 4; i++)
             {
                 var cpu = Math.Clamp(avgCpu + rng.Next(-variance, variance + 1), 0, 100);
@@ -479,7 +499,7 @@ private async Task SeedThinBaselineCpu(int avgCpu, int variance)
         var rng = new Random(42);
         foreach (var day in new[] { 7, 14 })
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 8; i++)
             {
                 var cpu = Math.Clamp(avgCpu + rng.Next(-variance, variance + 1), 0, 100);
@@ -495,7 +515,7 @@ private async Task SeedBaselinePerfmon(string counterName, long avgValue, int va
         var rng = new Random(42);
         for (int day = 1; day <= 14; day++)
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 4; i++)
             {
                 var value = Math.Max(0, avgValue + rng.Next(-variance, variance + 1));
@@ -511,7 +531,7 @@ private async Task SeedBaselineSessions(int avgConnections, int variance)
         var rng = new Random(42);
         for (int day = 1; day <= 14; day++)
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 4; i++)
             {
                 var count = Math.Max(1, avgConnections + rng.Next(-variance, variance + 1));
@@ -527,7 +547,7 @@ private async Task SeedBaselineQueryStats(long avgElapsed, int variance)
         var rng = new Random(42);
         for (int day = 1; day <= 14; day++)
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 4; i++)
             {
                 var elapsed = Math.Max(0, avgElapsed + rng.Next(-variance, variance + 1));
@@ -542,7 +562,7 @@ private async Task SeedBaselineWaits()
         await ExecuteSeedAsync("BEGIN TRANSACTION");
         for (int day = 1; day <= 14; day++)
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 4; i++)
                 await SeedWaitStatAsync(baseDay.AddMinutes(i * 3), "SOS_SCHEDULER_YIELD", 100);
         }
@@ -554,7 +574,7 @@ private async Task SeedBaselineMemory(double avgTotalServerMb, double targetMb)
         await ExecuteSeedAsync("BEGIN TRANSACTION");
         for (int day = 1; day <= 14; day++)
         {
-            var baseDay = _analysisStart.AddDays(-day);
+            var baseDay = SeedDayStart(day);
             for (int i = 0; i < 4; i++)
                 await SeedMemoryStatAsync(baseDay.AddMinutes(i * 3), avgTotalServerMb, targetMb);
         }

Lite/App.xaml.cs

@@ -196,6 +196,15 @@ private static string GetDefaultCsvSeparator()
         return System.Globalization.CultureInfo.CurrentCulture.NumberFormat.NumberDecimalSeparator == "," ? ";" : ",";
     }
 
+    /* Collection settings */
+    /* #2167: the Query Store history backfill (#2058) — fills gaps the live path never takes (a
+       first-contact tail, an outage hole, a freshly restored database's imported catalog) in bounded
+       background slices. Default ON. Turn it off when a heavy catch-up is costing the monitored server
+       more than the history is worth; live collection is unaffected and re-enabling resumes exactly
+       where the watermarks left off, so nothing is lost by pausing it. Darling's equivalent is a store
+       column (V58) because a headless service has no window to click. */
+    public static bool QueryStoreBackfillEnabled { get; set; } = true;
+
     /* System tray settings */
     public static bool MinimizeToTray { get; set; } = true;
 
@@ -779,6 +788,7 @@ only the enable flag and EU-region toggle are plain prefs. */
             if (root.TryGetProperty("smtp_recipients", out v)) SmtpRecipients = v.GetString() ?? "";
 
             if (root.TryGetProperty("analysis_enabled", out v)) AnalysisEnabled = v.GetBoolean();
+            if (root.TryGetProperty("query_store_backfill_enabled", out v)) QueryStoreBackfillEnabled = v.GetBoolean();
             if (root.TryGetProperty("analysis_notifications_enabled", out v)) AnalysisNotificationsEnabled = v.GetBoolean();
             if (root.TryGetProperty("analysis_interval_minutes", out v)) AnalysisIntervalMinutes = (int)Math.Clamp(v.GetInt64(), 5, 360);
             if (root.TryGetProperty("analysis_notify_severity", out v)) AnalysisNotifySeverity = Math.Clamp(v.GetDouble(), 0.0, 2.0);

Lite/Services/CollectionBackgroundService.cs

@@ -42,6 +42,11 @@ public class CollectionBackgroundService : BackgroundService
     /* #2058: the Query Store backfill tick — one byte-budgeted slice per server per due-tick, on
        Lite's IfDue ladder (the archival/retention/analysis idiom) rather than a separate loop. */
     private DateTime _lastQueryStoreBackfill = DateTime.MinValue;
+
+    /* #2167: last observed state of the backfill switch, so the log records each TRANSITION once instead of
+       once per idle tick. Starts true to match the setting's default — a deployment that never touches the
+       switch therefore logs nothing about it. */
+    private bool _queryStoreBackfillWasEnabled = true;
     private DateTime _lastRetentionTime = DateTime.UtcNow;
     private DateTime _lastAnalysisTime = DateTime.UtcNow;
     private DateTime _lastFindingsCleanupTime = DateTime.UtcNow;
@@ -194,6 +199,27 @@ property reads — negligible overhead at 1-minute cadence. */
     /// RemoteCollectorService.QueryStoreBackfill for the worker itself.</summary>
     private async Task RunQueryStoreBackfillIfDueAsync(CancellationToken stoppingToken)
     {
+        /* #2167: the off switch, checked BEFORE the due-time stamp so a disabled backfill does not quietly
+           consume its own schedule — flipping it back on runs on the next due tick rather than waiting out
+           an interval that elapsed while it was off. Read live from the setting (not captured), so the
+           Settings window takes effect without restarting Lite, matching Darling's store-reload behavior. */
+        if (!App.QueryStoreBackfillEnabled)
+        {
+            if (_queryStoreBackfillWasEnabled)
+            {
+                _queryStoreBackfillWasEnabled = false;
+                _logger?.LogInformation("Query Store backfill disabled in settings — idling; in-flight slices finish and no new ones start");
+            }
+
+            return;
+        }
+
+        if (!_queryStoreBackfillWasEnabled)
+        {
+            _queryStoreBackfillWasEnabled = true;
+            _logger?.LogInformation("Query Store backfill re-enabled in settings — resuming from the stored watermarks");
+        }
+
         if (DateTime.UtcNow - _lastQueryStoreBackfill < QueryStoreBackfillInterval)
         {
             return;

Lite/Windows/SettingsWindow.xaml

@@ -42,6 +42,7 @@
 
         <!-- Collection Control -->
         <Border Grid.Row="1" Background="{DynamicResource BackgroundLightBrush}" CornerRadius="4" Padding="12" Margin="0,0,0,12">
+            <StackPanel>
             <Grid>
                 <Grid.ColumnDefinitions>
                     <ColumnDefinition Width="*"/>
@@ -58,6 +59,16 @@
                 <Button Grid.Column="1" x:Name="PauseResumeButton" Content="Pause Co_llection"
                         Click="PauseResumeButton_Click" VerticalAlignment="Center"/>
             </Grid>
+            <!-- #2167: a COLLECTION control, so it lives beside Pause Collection rather than under
+                 Automated Analysis where it first landed (review catch) — the analysis engine has
+                 nothing to do with Query Store history. Mirrors where Darling puts it, next to plan
+                 capture in its collection group. -->
+            <CheckBox x:Name="QueryStoreBackfillCheckBox" Content="Fill Query Store history gaps in the background"
+                      Margin="0,10,0,0" Foreground="{DynamicResource ForegroundBrush}"/>
+            <TextBlock Text="Catches up Query Store history the live path never sees: a newly added server's recent tail, gaps left by an outage, and a freshly restored database's imported catalog. Runs in bounded background slices. Turn it off when a heavy catch-up is costing a monitored server more than the history is worth — live collection is unaffected, and re-enabling resumes exactly where it left off."
+                       FontSize="11" FontStyle="Italic" Foreground="{DynamicResource ForegroundMutedBrush}"
+                       Margin="20,2,0,0" TextWrapping="Wrap"/>
+            </StackPanel>
         </Border>
 
         <!-- MCP Server Settings -->

Lite/Windows/SettingsWindow.xaml.cs

@@ -526,6 +526,7 @@ private void LoadAlertSettings()
         };
         LogAlertDismissalsCheckBox.IsChecked = App.LogAlertDismissals;
         AnalysisEnabledCheckBox.IsChecked = App.AnalysisEnabled;
+        QueryStoreBackfillCheckBox.IsChecked = App.QueryStoreBackfillEnabled;
         AnalysisNotificationsCheckBox.IsChecked = App.AnalysisNotificationsEnabled;
         AnalysisIntervalBox.Text = App.AnalysisIntervalMinutes.ToString();
         AnalysisNotifySeverityBox.Text = App.AnalysisNotifySeverity.ToString("0.0", System.Globalization.CultureInfo.InvariantCulture);
@@ -617,6 +618,7 @@ make the gate impossible to turn back off once enabled. */
         App.MuteRuleDefaultExpiration = (MuteRuleDefaultExpirationCombo.SelectedItem as ComboBoxItem)?.Content?.ToString() ?? "24 hours";
         App.LogAlertDismissals = LogAlertDismissalsCheckBox.IsChecked == true;
         App.AnalysisEnabled = AnalysisEnabledCheckBox.IsChecked == true;
+        App.QueryStoreBackfillEnabled = QueryStoreBackfillCheckBox.IsChecked == true;
         App.AnalysisNotificationsEnabled = AnalysisNotificationsCheckBox.IsChecked == true;
         if (int.TryParse(AnalysisIntervalBox.Text, out var analysisInterval) && analysisInterval >= 5 && analysisInterval <= 360)
             App.AnalysisIntervalMinutes = analysisInterval;
@@ -693,6 +695,7 @@ make the gate impossible to turn back off once enabled. */
             root["mute_rule_default_expiration"] = App.MuteRuleDefaultExpiration;
             root["log_alert_dismissals"] = App.LogAlertDismissals;
             root["analysis_enabled"] = App.AnalysisEnabled;
+            root["query_store_backfill_enabled"] = App.QueryStoreBackfillEnabled;
             root["analysis_notifications_enabled"] = App.AnalysisNotificationsEnabled;
             root["analysis_interval_minutes"] = App.AnalysisIntervalMinutes;
             root["analysis_notify_severity"] = App.AnalysisNotifySeverity;