Community Health: monthly community health report system by functionzz · Pull Request #4336 · mozilla/pontoon

functionzz · 2026-07-25T00:39:06Z

This PR creates a community health alert system for PMs, and allows PMs see month-over-month changes to CHS at a glance. It is compiled monthly.

Fixes #4329.

codecov-commenter · 2026-07-25T00:43:13Z

Codecov Report

❌ Patch coverage is 94.73684% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.82%. Comparing base (ef4089a) to head (33af790).
⚠️ Report is 8 commits behind head on main.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mathjazz

Nice work!

Some high-level questions:

Why is the /insights page shown to and the notification sent to is_staff users , but the emails can be opted in by is_superuser users?
Why don't we automatically opt in all is_staff users also for receiving emails and drop the settings? This is an altert system afterall. Or at least opt them in by default, if we really want to keep the ability to opt out.
Have we tested the notification in the translate view, which has its own notification implementation?
The docstring says rows are "ranked by the size of that change", but the biggest gains come first and the worst regressions land at the bottom of the table. In an alert system, the regressions are the actionable half. Sort by abs(percentage) to match the docstring?
The new command needs to be documented on documentation/docs/dev/deployment.md. We should mention it silently depends on cron ordering - it reads the current month's snapshot — which collect_chs_snapshots creates on that same day.
We should also document the reporting feature for the Admins. Note that there's still #4338 (comment).

mathjazz · 2026-08-05T11:55:28Z


          {% if description and (description.startswith("Unreviewed suggestions") or notification.verb == "has reviewed suggestions" or notification.verb == "ignore") %}
-            <span class="description">{{ description|safe }}</span>
+            <div class="description">{{ description|safe }}</div>


What's the purpose of this change?

If we keep it as a span, if the description contains html elements, they will all be converted to text, which is not what we want.

The main point is now we have a system notification that automatically sends but also contains more complex html + css, which was not supported with the current code. All our previous system notifications were just one-liner spans.

Could we render this content under the existing .message class rather than restyling .description? The heading, paragraph, list and table styling already exists for .message — we'd keep the div, since block elements can't live in a span, but UserNotification.css wouldn't need the new block, and style.css wouldn't need the merged selectors either.

One consequence to weigh: the unreviewed-suggestions notification renders through the same component, so it would also pick up .message's padding and heading styling. Probably an improvement, but worth a look before and after.

mathjazz · 2026-08-05T11:55:40Z

What's the purpose of these changes?

Because I changed description to a div, I also want it to inherit css styling as well.

mathjazz · 2026-08-05T12:47:49Z

    assert mail.outbox[0].to == [user_a.contact_email]


+@pytest.mark.django_db


We should also add a test for the case when none of the locales passes a threshold.

flodolo · 2026-08-05T13:26:32Z

Why don't we automatically opt in all is_staff users also for receiving emails and drop the settings? This is an altert system afterall. Or at least opt them in by default, if we really want to keep the ability to opt out.

I don't think all superusers are PMs interested in this information, so they should have a way to opt out. I have the feeling the majority falls into this category, hence the opt-in suggestion to Jamie.

functionzz · 2026-08-07T17:45:29Z

Why is the /insights page shown to and the notification sent to is_staff users , but the emails can be opted in by is_superuser users?

I actually think it should be only reserved to is_staff users - I made the change before you reviewed, but it seems I didn't completely change options. ~~I need to change both user pool to be from is_staff.~~ Change complete.

Why don't we automatically opt in all is_staff users also for receiving emails and drop the settings? This is an alert system after all. Or at least opt them in by default, if we really want to keep the ability to opt out.

Just like the other email systems in Pontoon, I think we should keep the option to opt in given both the notification & email output very similar things.

Have we tested the notification in the translate view, which has its own notification implementation?

~~Not yet, will do that now.~~ Merged css + <span> -> <div>changes similar to django notifications.

The docstring says rows are "ranked by the size of that change", but the biggest gains come first and the worst regressions land at the bottom of the table. In an alert system, the regressions are the actionable half. Sort by abs(percentage) to match the docstring?

Sure. Implemented.

The new command needs to be documented on documentation/docs/dev/deployment.md. We should mention it silently depends on cron ordering - it reads the current month's snapshot — which collect_chs_snapshots creates on that same day. We should also document the reporting feature for the Admins. Note that there's still #4338 (comment).

Sure, I'll make the changes in this PR. Implemented.

…l test

…text

…k command logic

functionzz · 2026-08-10T17:48:52Z

I've been testing out the 2% threshold for CHS, and I'm beginning to think that is not sufficient, as some Common Voice locales may have deltas greater than 2% but these changes an insignificantly small. Perhaps changing to a 2 point threshold makes more sense?

mathjazz · 2026-08-10T21:20:45Z

I don't think all superusers are PMs interested in this information, so they should have a way to opt out. I have the feeling the majority falls into this category, hence the opt-in suggestion to Jamie.

OK. We should at least file an issue then to introduce a "PM" user role.

mathjazz

Thanks for the update!

Could you please unify the copy between the notification and the email and update the screenshot with the latest version?

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

mathjazz · 2026-08-12T11:27:11Z

I've been testing out the 2% threshold for CHS, and I'm beginning to think that is not sufficient, as some Common Voice locales may have deltas greater than 2% but these changes an insignificantly small. Perhaps changing to a 2 point threshold makes more sense?

How did we get to 2% in the first place? Have we looked at the old data?

functionzz · 2026-08-12T14:33:22Z

How did we get to 2% in the first place?

We didn't really discuss it, this was something I more or less added as a placeholder.

mathjazz

Left some notes to make the copy a bit more condensed.

…port.html Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

mathjazz

Sent copy through LLM and left some suggestions.

The rest looks good, great work!

mathjazz · 2026-08-24T18:31:57Z

+information on locale growth & decline in easily ingestible formats.
+
+## Insights dashboard
+


I would drop this section, which would allow us to keep the existing heading levels. 4 levels seems like a lot for a relatively short page.

mathjazz · 2026-08-24T18:40:07Z

+
+A team is included when it is enabled for at least one [key project](#key-projects), has a snapshot in both months, and its score changed by at least 2%. The threshold can be changed through the `MONTHLY_HEALTH_REPORT_CHS_THRESHOLD` environment variable.
+
+Staff users receive the report as a notification in Pontoon. To also receive it by email, enable `Monthly locale health report` in the `Email` section of your [settings](../localizer/users.md).


We link to the Settings page, but the new setting is not documented there.

mathjazz · 2026-08-24T20:03:16Z


          {% if description and (description.startswith("Unreviewed suggestions") or notification.verb == "has reviewed suggestions" or notification.verb == "ignore") %}
-            <span class="description">{{ description|safe }}</span>
+            <div class="description">{{ description|safe }}</div>


Could we render this content under the existing .message class rather than restyling .description? The heading, paragraph, list and table styling already exists for .message — we'd keep the div, since block elements can't live in a span, but UserNotification.css wouldn't need the new block, and style.css wouldn't need the merged selectors either.

One consequence to weigh: the unreviewed-suggestions notification renders through the same component, so it would also pick up .message's padding and heading styling. Probably an improvement, but worth a look before and after.

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

…rt_notifications_and_emails.py Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

mathjazz

Looks great, thanks!

functionzz force-pushed the community_health_alert_system branch 2 times, most recently from 96979a9 to e6cf8c9 Compare August 4, 2026 13:28

functionzz requested a review from mathjazz August 4, 2026 15:46

mathjazz mentioned this pull request Aug 5, 2026

Spec: Introduce Community Health Score panel #4367

Merged

mathjazz reviewed Aug 5, 2026

View reviewed changes

functionzz and others added 10 commits August 7, 2026 14:22

add monthly community health report system

2f716b2

add tests for notifications, reports

1c09b35

add email send functionality, rename files

903f7a6

switch threshold to percentage, edit docstrings & templates, add emai…

d111095

…l test

make migration (again)

929fe18

make migration (again) (again)

b184e6e

change to staff status

333f6ac

fix test

e6f7ddb

make migration (again) (again) (again)

5067477

switch is_superuser to is_staff for email & notification sending

b9a755c

functionzz force-pushed the community_health_alert_system branch from 5b5bde8 to b9a755c Compare August 7, 2026 18:22

functionzz added 6 commits August 7, 2026 14:29

make migration (again) (again) (again) (again)

617f874

make format

4296f7e

add deployment documentation, sort locale rows by abs(%)

6bbf849

add deployment doc + include key projects documentation

c5a088b

add translate app noti styles + add back white color scheme for noti …

6eb5002

…text

allow noti and email to send even if no locales cross threshold, twea…

ba16471

…k command logic

functionzz requested a review from mathjazz August 10, 2026 17:47

mathjazz reviewed Aug 10, 2026

View reviewed changes

Comment thread pontoon/contributors/templates/contributors/settings.html Outdated

Update pontoon/contributors/templates/contributors/settings.html

3941e01

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

functionzz added 2 commits August 12, 2026 01:30

equate email and notification

b526f34

make format

f37c8fb

functionzz requested a review from mathjazz August 12, 2026 05:41

functionzz and others added 2 commits August 21, 2026 09:57

Merge branch 'main' into community_health_alert_system

65d57f5

@functionzz make migration (again) (again) (again) (again) {again}

c12a901

mathjazz reviewed Aug 21, 2026

View reviewed changes

functionzz and others added 6 commits August 21, 2026 10:37

Update pontoon/messaging/templates/messaging/emails/monthly_health_re…

8e18c42

…port.html Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update pontoon/messaging/templates/messaging/emails/monthly_health_re…

0048c3f

…port.html Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update pontoon/messaging/templates/messaging/emails/monthly_health_re…

a646910

…port.html Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update pontoon/messaging/templates/messaging/emails/monthly_health_re…

97de924

…port.html Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

copy email changes to notif

658208f

edit tests

9e771eb

functionzz requested a review from mathjazz August 21, 2026 14:49

mathjazz reviewed Aug 24, 2026

View reviewed changes

functionzz and others added 11 commits August 25, 2026 09:05

Update documentation/docs/admin/insights.md

6983e9f

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update documentation/docs/admin/insights.md

e421459

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update documentation/docs/admin/insights.md

581929e

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update pontoon/settings/base.py

288c65a

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update pontoon/messaging/management/commands/send_monthly_health_repo…

3dd33fa

…rt_notifications_and_emails.py Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update documentation/docs/admin/insights.md

46aabf8

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

Update documentation/docs/admin/insights.md

b0e9db9

Co-authored-by: Matjaž Horvat <matjaz.horvat@gmail.com>

change notification class to message, remove redundant css

66201d5

fix misaligned notification

ef4c01d

fix alignment for translate app noti

7ecaaf0

rename monthly locale health report to monthly health report

33af790

functionzz requested a review from mathjazz August 25, 2026 21:44

mathjazz approved these changes Aug 26, 2026

View reviewed changes

mathjazz merged commit c7ba26f into mozilla:main Aug 26, 2026
12 checks passed

		assert mail.outbox[0].to == [user_a.contact_email]


		@pytest.mark.django_db

		information on locale growth & decline in easily ingestible formats.

		## Insights dashboard


		A team is included when it is enabled for at least one [key project](#key-projects), has a snapshot in both months, and its score changed by at least 2%. The threshold can be changed through the `MONTHLY_HEALTH_REPORT_CHS_THRESHOLD` environment variable.

		Staff users receive the report as a notification in Pontoon. To also receive it by email, enable `Monthly locale health report` in the `Email` section of your [settings](../localizer/users.md).

Uh oh!

Conversation

functionzz commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mathjazz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flodolo commented Aug 5, 2026

Uh oh!

functionzz commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

functionzz commented Aug 10, 2026

Uh oh!

mathjazz commented Aug 10, 2026

Uh oh!

mathjazz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mathjazz commented Aug 12, 2026

Uh oh!

functionzz commented Aug 12, 2026

Uh oh!

mathjazz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mathjazz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mathjazz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

functionzz commented Jul 25, 2026 •

edited

Loading

codecov-commenter commented Jul 25, 2026 •

edited

Loading

functionzz commented Aug 7, 2026 •

edited

Loading