PIR: Forward extras on extracted profiles - #9353
Merged
Merged
Conversation
8 tasks
noisysocks
marked this pull request as draft
July 31, 2026 01:22
noisysocks
force-pushed
the
randerson/pir-extras-passthrough
branch
from
August 3, 2026 05:03
5b4c506 to
38f5db8
Compare
noisysocks
changed the title
noisysocks
marked this pull request as ready for review
August 3, 2026 05:10
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as low quality.
This comment was marked as low quality.
landomen
requested changes
Aug 5, 2026
landomen
left a comment
landomen
left a comment
Contributor
There was a problem hiding this comment.
@noisysocks The changes generally look good, and PIR state was kept after installing updated build 👍
Left a few comments that we need to address, most importantly the transaction one since it can cause data inconsistencies.
noisysocks
force-pushed
the
randerson/pir-extras-passthrough
branch
from
August 6, 2026 00:46
492df89 to
7642990
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Collaborator
Author
|
@landomen: Addressed / responded to your notes. I’m no Android dev so leaning on asking what you’d suggest more than I ordinarily would 😅 |
noisysocks
commented
Aug 6, 2026
landomen
reviewed
Aug 10, 2026
landomen
left a comment
landomen
left a comment
Contributor
There was a problem hiding this comment.
@noisysocks I see unit tests are failing because of an unrelated issue. Could you try to rebase the branch with latest develop to see if it resolves it?
Adds an open extras map to extracted profiles and to their addresses. Native never looks inside it: it reads extras off the extract message, stores them, and hands them back verbatim when an opt-out form is filled. A new field can then be added in the broker config alone, rather than needing a change and a release on all three platforms. - Parse profile-level and address-level extras from the extract response, strictly as Map<String, String>, so malformed values fail loudly through the existing message-parsing error path. - Persist them: new extras column on pir_extracted_profiles via a v17 auto migration, with address extras carried inside the existing addresses JSON. - Send addresses, extras, age, phoneNumbers, relatives and identifier to fillForm. Previously only name, profileUrl, fullName and email were sent, so addresses were dropped entirely. - Refresh stored profiles on re-scan rather than ignoring them. Extras are merged, so a key missing from a later scrape keeps the value already stored and a pending opt-out doesn't lose fields when a broker changes its markup. - Exclude extras from profile identity comparisons, so shipping a config that starts scraping a new field cannot make stored records look removed. Tech design: https://app.asana.com/1/137249556945/project/481882893211075/task/1216773212741757 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wrap the read-merge-write in saveExtractedProfiles in a Room transaction so a concurrent edit - a profile being deprecated from the dashboard, say - can't be silently overwritten by refreshedWith carrying over stale locally owned fields. Move ExtractedProfileStorageKey to the bottom of the class. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WKNJoD8pHnNpgzNo1SreM
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Storage matched profiles on five fields while removal and reappearance detection compared all thirteen modelled fields. A broker re-listing the same record with a changed age or address refreshed the stored row but looked absent to reconciliation, marking the opt-out REMOVED and then reverting it to REQUESTED on the next scan, firing a spurious reappearance pixel. Move the key to PirUtils as ExtractedProfileKey / toKey() and use it for both, so a scrape that refreshes a row in place also counts as that row still being present on the broker. Also unwrap the explicit Runnable in the saveExtractedProfiles transaction. https://app.asana.com/1/137249556945/task/1216773212741753 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
noisysocks
force-pushed
the
randerson/pir-extras-passthrough
branch
from
August 11, 2026 00:04
fff8971 to
d6e59b3
Compare
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d6e59b3. Configure here.
The continuation lines in matches() were indented 12 spaces where ktlint wants 8, failing spotlessKotlinCheck. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ExtractProfileSelectors was a closed data class holding the 12 selector fields we know about, so Moshi dropped every other key when parsing a broker config. Since we re-serialise the parsed action to push it to C-S-S, those keys never reached the script. That defeats the point of config-declared extras: C-S-S scrapes any field name it doesn't recognise into the extracted profile's extras, so a broker config could not add one without an app release. It also dropped knobs on the fields we do model - attribute, source, and the nested city / state sub-selectors - which C-S-S supports and we were silently discarding. Nothing on our side reads any of this, so the profile block is now carried as raw json and forwarded verbatim. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
landomen
approved these changes
Aug 12, 2026
landomen
left a comment
landomen
left a comment
Contributor
There was a problem hiding this comment.
Left few minor comments, otherwise I ran a scan on this branch and on develop and got same results so I think we're good 👍
Brings alternativeNames in line with the other extracted profile fields we already pass to fillForm, so a broker config can reference it without a native release. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task/Issue URL:
https://app.asana.com/1/137249556945/task/1216773212741753
Tech Design URL (if applicable):
https://app.asana.com/1/137249556945/project/481882893211075/task/1216773212741757
API Proposals URL(s) (if applicable):
Description
Support
extrasin extracted profiles and addresses, pass them along tofillForm, and update stored profiles with the freshly scraped profile. This lessens the need for an app release when an opt out form requires a new field we don't have.Steps to test this PR
cyberbackgroundchecks.comand replace the JSON with: https://dub.duckduckgo.com/duckduckgo/dbp-api/blob/2fb2bb29f5d3f67bc6c70098ca36db7ca4a100b9/dbp-json/data/json/cyberbackgroundchecks.com.jsonNote
Medium Risk
Touches PIR local storage, scan refresh semantics, and opt-out identity matching; changes are well-tested but incorrect merge or key logic could affect pending opt-outs or removal detection.
Overview
Adds
extrason extracted profiles and addresses so broker configs can scrape arbitrary fields without a native release, and passes the fuller profile through to C-S-S forfillForm.On repeat scans,
saveExtractedProfilesrefreshes rows that already match the DB unique key (profile query, broker, name, URL, identifier) instead of ignoring them, merging extras so keys missing from a new scrape keep stored values. Profile identity for removals/reappearances uses a narrower key andmatchesignores address extras so new scraped fields don’t look like a different person.ExtractProfileSelectorsbecomes a passthrough map so extract profile config isn’t dropped at parse time. Room bumps to v17 with anextrascolumn, Moshi map converters, and migration tests.Reviewed by Cursor Bugbot for commit 71d26d7. Bugbot is set up for automated code reviews on this repo. Configure here.