feat(Environments): Track first evaluation with a new Onboarding interface by emyller · Pull Request #8063 · Flagsmith/flagsmith

emyller · 2026-07-21T19:53:55Z

I have read the Contributing Guide.
I have added information to docs/ if required so people know about the feature.
I have filled in the "Changes" section below.
I have filled in the "How did you test this code" section below.

Changes

Closes #7623

New /api/v1/environments/{api_key}/onboarding-status/ API:
- GET obtains first evaluation data.
- PUT records first evaluation, or skips.
The environment document (for DynamoDB) gets a new attribute onboarding_pending.
Core /api/v1/flags/ also records first evaluation.

Review effort: 3/5

How did you test this code?

Unit tests.

vercel · 2026-07-21T19:54:00Z

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	Jul 22, 2026 9:45pm
flagsmith-frontend-preview	Ignored	Preview	Jul 22, 2026 9:45pm
flagsmith-frontend-staging	Ignored	Preview	Jul 22, 2026 9:45pm

coderabbitai · 2026-07-21T19:54:02Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds nullable first-evaluation fields to environments, with clone reset behaviour and migration support. Introduces an onboarding status API for reading and recording evaluation state, including validation, idempotent persistence, document writes, logging, and tests. SDK user-agent detection now records the first evaluation during flag requests. Engine and DynamoDB mappings expose onboarding_pending, while SDK documents exclude it. Observability documentation lists the new onboarding events.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Zaimwa9 · 2026-07-21T20:06:19Z

@themis-blindfold review

themis-blindfold · 2026-07-21T20:12:29Z

⚖️ Themis judgement: 🟠 Fix before merge

New /onboarding-status/ endpoint to track when an environment's flags are first evaluated by an SDK. The GET path has an unhandled DoesNotExist that surfaces as a 500, and a test assertion is checking the wrong keyword argument — likely the cause of the CI failures visible on this commit.


🎯 Correctness	3/5
🧪 Test coverage	3/5
📐 Code quality	4/5
🚀 Product impact	3/5

🟠 Majors

api/environments/onboarding/views.py:27 — GET returns 500 on non-existent environment API key (unhandled DoesNotExist).
api/tests/unit/environments/onboarding/test_unit_environments_views.py:161 — test asserts api_key= but the task passes environment_id=; Observed: API Unit Tests (3.11, 3.13) failed in CI.

🧹 Nits

api/environments/onboarding/tasks.py:31 — check-then-update on first_evaluated_at is not atomic; two concurrent tasks could both write. A filter(..., first_evaluated_at__isnull=True).update(...) would close the window.

📝 Walkthrough

Model & migration — Adds nullable first_evaluated_at and first_evaluated_sdk_label fields to Environment (and its historical model).
Onboarding endpoint — New unauthenticated GET/PUT at /{api_key}/onboarding-status/ for SDKs to report first evaluation; PUT enqueues a task.
Task — record_environment_first_evaluation sets the fields once and refreshes the environment document in DynamoDB.
Engine model & mappers — Adds onboarding_pending boolean to the engine model, derived from first_evaluated_at is None, excluded from the SDK document.
SDK mapper refactor — map_environment_to_sdk_document simplified; SDK_DOCUMENT_EXCLUDE changed from list to set.
Docs & OpenAPI — Events catalogue updated; OpenAPI spec extended.

🧪 How to verify

Fix the test assertion (environment_id=environment.id), re-run the unit test file, confirm all pass.
Add get_object_or_404 (or a try/except returning 404) to the GET handler, then curl the endpoint with an invalid API key and confirm a 404.
Run the existing DynamoDB mapper tests to confirm onboarding_pending round-trips correctly.
Hit the PUT endpoint twice for the same environment and confirm the second call is a no-op (check logs for environment.already_evaluated).

Automate: a test for GET with an unknown API key asserting 404.

Product take: Solid onboarding signal — knowing when and which SDK first evaluates an environment lets the dashboard guide new users through setup. Meaningful for activation tracking; moderate product impact.

🧭 Assumptions & unverified claims

Assumed the task_processor_synchronously autouse fixture (conftest line 892) causes .delay() to run inline in tests, making the "sync" test group exercise the real task code. Could not import the library in this environment to confirm.
The OpenAPI spec lists 200 for the PUT response; the actual code returns 204. Assumed this is an auto-generation artefact rather than an intentional contract.

First evaluation is a write-once operation, but that test assertion only gets one chance too — and it picked the wrong keyword. · reviewed at 9c3dfa7

github-actions · 2026-07-21T20:18:36Z

Docker builds report

Image	Build Status	Security report
`ghcr.io/flagsmith/flagsmith-e2e:pr-8063`	Finished ✅	Skipped
`ghcr.io/flagsmith/flagsmith-api-test:pr-8063`	Finished ✅	Skipped
`ghcr.io/flagsmith/flagsmith-frontend:pr-8063`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith-api:pr-8063`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith:pr-8063`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith-private-cloud:pr-8063`	Finished ✅	Results ✅

coderabbitai

Actionable comments posted: 2

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 616f3493-5fbb-4b3b-8c20-3b1649411858

📥 Commits

Reviewing files that changed from the base of the PR and between 94ac7af and 9c3dfa7.

📒 Files selected for processing (17)

api/environments/migrations/0038_add_first_evaluated_fields.py
api/environments/models.py
api/environments/onboarding/__init__.py
api/environments/onboarding/serializers.py
api/environments/onboarding/tasks.py
api/environments/onboarding/urls.py
api/environments/onboarding/views.py
api/environments/urls.py
api/tests/unit/environments/onboarding/__init__.py
api/tests/unit/environments/onboarding/test_unit_environments_views.py
api/tests/unit/util/mappers/test_unit_mappers_dynamodb.py
api/tests/unit/util/mappers/test_unit_mappers_engine.py
api/util/engine_models/environments/models.py
api/util/mappers/engine.py
api/util/mappers/sdk.py
docs/docs/deployment-self-hosting/observability/_events-catalogue.md
openapi.yaml

github-actions · 2026-07-21T20:25:07Z

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #18642 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

3 passed

Details

3 tests across 3 suites
35.9 seconds
b0ef881
🔄 Run: #18642 (attempt 1)

🗂️ Previous results

✅ private-cloud · depot-ubuntu-latest-16 — run #18642 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
32.8 seconds
b0ef881
🔄 Run: #18642 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #18643 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
39.5 seconds
2197c0b
🔄 Run: #18643 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #18643 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
41.2 seconds
2197c0b
🔄 Run: #18643 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #18642 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
44.8 seconds
b0ef881
🔄 Run: #18642 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #18642 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
33.7 seconds
b0ef881
🔄 Run: #18642 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-16 — run #18641 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
36.2 seconds
1fedc19
🔄 Run: #18641 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #18641 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

2 passed

Details

2 tests across 2 suites
37.6 seconds
1fedc19
🔄 Run: #18641 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #18640 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

2 passed

Details

2 tests across 2 suites
36.7 seconds
1e15195
🔄 Run: #18640 (attempt 1)

✅ private-cloud · depot-ubuntu-latest-16 — run #18640 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
33.8 seconds
1e15195
🔄 Run: #18640 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #18641 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

2 passed

Details

2 tests across 2 suites
37.1 seconds
1fedc19
🔄 Run: #18641 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #18640 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
44.3 seconds
1e15195
🔄 Run: #18640 (attempt 1)

coderabbitai

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

api/environments/onboarding/views.py (1)

34-48: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

Require an authenticated principal for onboarding updates.

This endpoint accepts anonymous writes and treats the URL API key as sufficient authority. Client SDK keys can be exposed to end users, allowing anyone with one to falsely complete onboarding and choose the recorded SDK label. Protect PUT with the existing environment-management permission model; retain a separate explicitly authorised skip flow if needed.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f8452941-6950-4da6-871d-94518549d663

📥 Commits

Reviewing files that changed from the base of the PR and between 9c3dfa7 and 01064ba.

📒 Files selected for processing (9)

api/app_analytics/constants.py
api/app_analytics/mappers.py
api/environments/models.py
api/environments/onboarding/tasks.py
api/environments/onboarding/views.py
api/features/views.py
api/tests/unit/environments/onboarding/test_unit_environments_views.py
api/tests/unit/environments/test_unit_environments_models.py
api/tests/unit/features/test_unit_features_views.py

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a8585923-95e5-4652-ae1e-5a23652cdafe

📥 Commits

Reviewing files that changed from the base of the PR and between 01064ba and e628afd.

📒 Files selected for processing (1)

openapi.yaml

khvn26

Looks good on the whole — two questions (which are essentially one question) before I approve, and a nit.

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b68cfc2b-c1bf-4237-8955-6a0fc72aeb9b

📥 Commits

Reviewing files that changed from the base of the PR and between e628afd and 1fedc19.

📒 Files selected for processing (11)

api/environments/onboarding/services.py
api/environments/onboarding/urls.py
api/environments/onboarding/views.py
api/features/views.py
api/tests/conftest.py
api/tests/test_helpers.py
api/tests/unit/environments/onboarding/test_unit_environments_views.py
api/tests/unit/features/test_unit_features_views.py
api/tests/unit/onboarding/conftest.py
api/tests/unit/onboarding/test_views.py
docs/docs/deployment-self-hosting/observability/_events-catalogue.md

💤 Files with no reviewable changes (1)

api/tests/unit/onboarding/conftest.py

khvn26

LGTM — the remaining conversation is non-blocking.

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

api/environments/onboarding/views.py (1)

18-20: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Require authentication for the PUT endpoint.

The view explicitly disables authentication and permissions, so any caller possessing only the environment_api_key can submit a valid SDK label. Because persistence is first-write-wins, this allows onboarding data to be poisoned before the legitimate SDK evaluates the environment. Apply the project’s standard authenticated permission to PUT, or split the public GET and authenticated PUT into separate views.

Also applies to: 27-35

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 69abd0de-ef75-4bda-93d3-1c75de70a4c6

📥 Commits

Reviewing files that changed from the base of the PR and between 1fedc19 and b0ef881.

📒 Files selected for processing (3)

api/environments/onboarding/urls.py
api/environments/onboarding/views.py
api/tests/unit/environments/onboarding/test_unit_environments_views.py

💤 Files with no reviewable changes (1)

api/tests/unit/environments/onboarding/test_unit_environments_views.py

khvn26

🚢

emyller added 6 commits July 21, 2026 11:51

red stuff

99c6e79

let there be data

24b67eb

green stuff

4cf4b70

go international 🌐

3eb3773

🪵

7922b5f

nah

772ad1b

emyller self-assigned this Jul 21, 2026

github-actions Bot added api Issue related to the REST API docs Documentation updates feature New feature or request and removed docs Documentation updates labels Jul 21, 2026

themis-blindfold Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread api/environments/onboarding/views.py Outdated

Comment thread api/tests/unit/environments/onboarding/test_unit_environments_views.py Outdated

Comment thread api/environments/onboarding/services.py

Zaimwa9 marked this pull request as ready for review July 21, 2026 20:16

Zaimwa9 requested review from a team as code owners July 21, 2026 20:16

Zaimwa9 requested review from khvn26 and removed request for a team July 21, 2026 20:16

github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Jul 21, 2026

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread api/environments/models.py

Comment thread openapi.yaml Outdated

Zaimwa9 marked this pull request as draft July 21, 2026 20:24

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread api/features/views.py

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread openapi.yaml

flagsmith-engineering Bot assigned khvn26 Jul 21, 2026

khvn26 reviewed Jul 22, 2026

View reviewed changes

Comment thread api/environments/onboarding/views.py Outdated

Comment thread api/features/views.py Outdated

Comment thread api/environments/onboarding/views.py

emyller force-pushed the feat/yay-first-eval branch from e628afd to 1e15195 Compare July 22, 2026 20:20

emyller added 2 commits July 22, 2026 17:21

wait it

9f6b6be

smithers-only

1fedc19

emyller force-pushed the feat/yay-first-eval branch from 1e15195 to 1fedc19 Compare July 22, 2026 20:21

github-actions Bot added feature New feature or request docs Documentation updates and removed feature New feature or request docs Documentation updates labels Jul 22, 2026

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread api/features/views.py

khvn26 previously approved these changes Jul 22, 2026

View reviewed changes

tweak api visibility

b0ef881

emyller dismissed khvn26’s stale review via b0ef881 July 22, 2026 21:43

github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Jul 22, 2026

chore: Update documentation artefacts

2197c0b

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

github-actions Bot added the docs Documentation updates label Jul 22, 2026

khvn26 approved these changes Jul 22, 2026

View reviewed changes

emyller merged commit 3c9cc97 into main Jul 22, 2026
34 checks passed

emyller deleted the feat/yay-first-eval branch July 22, 2026 21:59

flagsmith-engineering Bot mentioned this pull request Jul 22, 2026

chore(main): release 2.256.0 #8058

Open

Uh oh!

Conversation

emyller commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

How did you test this code?

Uh oh!

vercel Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Uh oh!

Zaimwa9 commented Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

themis-blindfold Bot commented Jul 21, 2026

⚖️ Themis judgement: 🟠 Fix before merge

🟠 Majors

🧹 Nits

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docker builds report

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-16)

Details

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

Details

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

Details

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

khvn26 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

khvn26 left a comment

emyller commented Jul 21, 2026 •

edited

Loading

vercel Bot commented Jul 21, 2026 •

edited

Loading

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading