UN-3973 [DEV] Cut dashboard cron DB time by deriving monthly metrics from the daily tier

[kirtimanmishrazipstack]

What

• event_metrics_monthly is now derived by rolling up event_metrics_daily, instead of re-querying source tables per organisation.
• The per-run source window for the daily tier drops from ~32–62 days to 2 days.
• A once-daily reconciliation pass reruns the same task at a 7-day window to repair gaps left by cron downtime.

Why

_aggregate_single_metric and _aggregate_llm_combined widened their DAY-granularity query to monthly_start — the first of the previous calendar month — so monthly buckets could be summed in Python from the same rows, saving a third query per metric. The cost was that every run re-read 32–62 days of source data for each of 38 organisations, 96 times a day.

Six of the eight cron queries are index-backed (11–107 ms avg), so their cost is proportional to that range. Per the analysis on UN-3883, ~460 s per 6 h is recoverable by narrowing it, with no schema change.

Two supporting facts make the rollup safe: event_metrics_daily retains 365 days against a 2-month monthly window (11 MB / 25 k rows), and the code already derives monthly by summing DAY buckets — this only moves where the sum happens, not what it sums.

The 2-day window is sized on measurement, not guesswork: over 30 days and 405,951 terminal rows on the read replica, the worst created_at → terminal-status lag was 2 h 16 m, with zero rows beyond 6 h. It is bounded by FILE_PROCESSING_TASK_TIME_LIMIT (2 h) and the 2.5 h stuck-execution reaper, so 2 days leaves ~21× margin.

This will not move the top-line 55-minute figure. It cannot touch get_documents_processed (44% of cron time) or get_failed_pages (43%) — those are UN-3972's. Measure it against §4 rows 5, 6, 7, 10 and 14.

How

All changes are in backend/dashboard_metrics/.

• Constants — DASHBOARD_SOURCE_WINDOW_DAYS = 2, DASHBOARD_RECONCILE_WINDOW_DAYS = 7, DASHBOARD_ACTIVE_ORG_LOOKBACK_DAYS = 7.

• aggregate_metrics_from_sources(source_window_days=2) threads the window through to _run_aggregation. The reconciliation pass is the same task at a different bound, so it needs no separate code path — just a second PeriodicTask row.

• _aggregate_single_metric / _aggregate_llm_combined lost their monthly_start and monthly_agg parameters. The DAY query now binds daily_start, and the if day_ts >= daily_start guard is gone — it was a Python filter over rows the query had already fetched, and the bind value does that work now. Still 2 queries per metric, just a narrow one.

• _bulk_upsert_monthly → _rollup_monthly_from_daily(month_start) — one ORM aggregate (TruncMonth + Sum) over event_metrics_daily, called once after the org loop for all organisations. The monthly tier stops scaling with tenant count.

  Two details worth review attention: the grouping matches unique_monthly_metric — (organization, month, metric_name, project, tag) — exactly, and metric_type is aggregated (Min) rather than grouped, because it is not part of that constraint. Grouping on it would let a metric whose type ever changed mid-month produce two rows on one key, which ON CONFLICT DO UPDATE rejects outright. The Min is aliased mtype because Django refuses an annotation named after a model field.

• The 15-line January special case for monthly_start collapsed to two lines using the existing _truncate_to_month helper, which handles the year rollover on its own.

• The active-org prefilter is decoupled from daily_start and pinned at 7 days — see the breakage section below.

Chose the ORM over the raw INSERT … SELECT sketched on the ticket: identical arithmetic, no string-built SQL, and it reuses the existing _base_manager convention for Celery context.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why.

Three real behaviour changes, all deliberate:

1. Monthly now inherits gaps in event_metrics_daily. The 62-day source window used to re-derive monthly from source tables, so it incidentally self-healed the monthly tier after cron downtime. Post-change, monthly is only as complete as the daily tier, and the reconciliation pass heals 7 days back. Downtime longer than 7 days needs the backfill_metrics management command. This is the trade the ticket accepts by design.

2. Source deletions stop self-correcting beyond the window. Deleting a workflow cascades to its executions and file executions. Pre-change, monthly was re-derived from source over 62 days, so that correction propagated. Post-change, monthly sums the daily tier and the daily tier keeps the stale bucket, so only the last 2 days — 7 on the reconciliation pass — self-correct. Older buckets stay as they were until a backfill_metrics run. The durable fix is invalidation when the workflow is deleted, not a wider scan in the cron; raised by Greptile and answered in-thread.

3. Lock contention on the reconciliation pass. Both schedules share AGGREGATION_LOCK_KEY, so a reconciliation pass that fires while a 15-minute run is in flight will skip that day (~10% of days). Nothing is lost — the next day's pass still covers 7 days back — unless a collision and multi-day downtime coincide. Judged not worth a retry branch.

What is not at risk: every dashboard metric filters and buckets on the same column (get_hitl_completions uses approved_at for both, the rest use created_at), so a row can only ever be counted in the bucket of the timestamp it is filtered by. A 2-day window therefore cannot silently drop a late-arriving row into an uncounted bucket.

One near-miss caught during implementation and fixed here: the active-org prefilter reused daily_start, so narrowing that variable would have silently narrowed the prefilter too. Since get_hitl_completions filters approved_at, an org approving today a document processed five days ago would have fallen out of active_org_ids and lost the metric. The prefilter now has its own constant and stays at 7 days — its own 1,849 ms cost is UN-3974's problem, not this PR's.

Database Migrations

• dashboard_metrics/0004_add_reconciliation_task.py — data migration only, no schema change. Creates one django_celery_beat PeriodicTask named dashboard_metrics_reconcile_source_window (04:00 UTC, queue dashboard_metric_events, kwargs={"source_window_days": 7}), chosen to sit clear of the existing 02:00 and 03:00 cleanup tasks. Uses update_or_create, so it is safe to re-run; the reverse deletes the row by name. 0002_setup_periodic_tasks.py is untouched.

Env Config

• None.

Relevant Docs

• UN-3883 analysis §6.2, §6.3, §7 step 1, §8 (revised 2026-08-12).

Related Issues or PRs

• Parent: UN-3883 — Analyze and optimize dashboard cron queries causing high DB load
• Siblings: UN-3972 (indexes), UN-3974 (schedule split + prefilter)
• Raised against UN-3883-Optimize-DB-cron-queries-causing-high-DB-load, not main.

Dependencies Versions

• None.

Notes on Testing

backend/dashboard_metrics/tests/test_tasks.py — 18 pass (9 pre-existing, 9 new). New coverage:

• _rollup_monthly_from_daily sums daily rows into the right month bucket
• month boundary — rows spanning the 1st land in two separate monthly rows with no bleed
• rows older than month_start are excluded
• rerunning the rollup overwrites rather than doubles (the ON CONFLICT path)
• a metric whose metric_type differs across days in one month yields one row, not a crash
• an empty daily tier is a no-op
• the per-run and reconciliation windows each bound the daily query correctly, and the task forwards its kwarg

Also verified manually against a local Postgres: the migration applies and all four PeriodicTask rows read back correct; makemigrations --check reports no model drift; and an end-to-end _run_aggregation over the real source tables returned errors: 0 with monthly.start = 2026-07-01 and daily.start seven days back for the reconciliation pass.

Screenshots

N/A — backend only.

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KGZBF68CShem3pbUJM2tBc

[greptile-apps]

Greptile Summary

The PR narrows recurring source aggregation to two days, adds a seven-day daily reconciliation task, and derives monthly dashboard metrics from the retained daily tier.

• Adds the 04:00 UTC reconciliation schedule through a data migration.
• Refactors per-organization hourly and daily collection around configurable source windows.
• Rebuilds monthly rows from grouped daily metrics and atomically removes obsolete keys.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
backend/dashboard_metrics/migrations/0004_add_reconciliation_task.py	Adds an idempotent daily Celery Beat reconciliation task with a reversible data migration.
backend/dashboard_metrics/tasks.py	Narrows source windows, separates organization collection, and derives monthly metrics from daily aggregates.
backend/dashboard_metrics/tests/test_tasks.py	Covers monthly grouping, month boundaries, overwrite and cleanup behavior, and configurable source windows.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Beat[15-minute Celery schedule] --> Aggregate[Aggregate source metrics]
    Reconcile[Daily 7-day reconciliation] --> Aggregate
    Aggregate --> Hourly[(Hourly metrics)]
    Aggregate --> Daily[(Daily metrics)]
    Daily --> Rollup[Monthly rollup]
    Rollup --> Monthly[(Monthly metrics)]

Loading

_{Reviews (4): Last reviewed commit: "UN-3973 Address Sonar and Greptile revie..." | Re-trigger Greptile}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[kirtimanmishrazipstack]

@greptile-apps please review