We still spend significant compute to scan segment membership counts every 6h for identities that weren't necessarily updated.
Acceptance criteria
- An
environment_id, maxState(inserted_at) watermark is materialised against the IDENTITIES table.
refresh_all_segment_counts only fans out segment membership count refresh tasks for projects which environment watermarks changed since the last query.- The watermark query complexity is not derived from total number of identities.
We still spend significant compute to scan segment membership counts every 6h for identities that weren't necessarily updated.
Acceptance criteria
environment_id, maxState(inserted_at)watermark is materialised against theIDENTITIEStable.refresh_all_segment_countsonly fans out segment membership count refresh tasks for projects which environment watermarks changed since the last query.