[JCC] - Add custom CloudWatch metrics for SSN corrections

backend/compact-connect/docs/README.md

@@ -62,8 +62,8 @@ leave the field entirely empty. If some of your licenses are missing a required
 | middleName | Provider's middle name (optional) | String (max 100 chars) | Robert |
 | npi | National Provider Identifier (optional) | 10-digit number | 1234567890 |
 | phoneNumber | Provider's phone number (optional) | [ITU-T E.164 format](https://www.itu.int/rec/T-REC-E.164-201011-I/en) (must include country code, no spaces or dashes) | +12025550123 |
-| previousSSN | The incorrect Social Security Number previously uploaded for this license (optional). Provide this along with the corrected `ssn` to fix a license record that was uploaded with the wrong SSN: the system moves the license record and any privileges purchased against it over to the provider associated with the corrected SSN. If the corrected license was the only license under the incorrect SSN, that duplicate provider account is removed and the practitioner is emailed to register again. | Format: XXX-XX-XXXX | 123-45-6789 |
 | suffix | Provider's name suffix (optional) | String (max 100 chars) | Jr. |
+| previousSSN | The incorrect Social Security Number previously uploaded for this license (optional). **This field is a last resort** for cases where a state has already uploaded incorrect SSN information that resulted in practitioners purchasing privileges or other actions being taken against the license (e.g., encumbrances, investigations, etc.) under the incorrect SSN, and should be avoided whenever possible — see [Can we upload the same licenses multiple times? What if their information changes?](#can-we-upload-the-same-licenses-multiple-times-what-if-their-information-changes) before using it. Provide this along with the corrected `ssn` to fix a license record that was uploaded with the wrong SSN: the system moves the license record and any privileges purchased against it over to the provider associated with the corrected SSN. If the corrected license was the only license under the incorrect SSN, that duplicate provider account is removed and the practitioner is emailed to register again. | Format: XXX-XX-XXXX | 123-45-6789 |
 ** This field is required by compact commission rule, however, to avoid making a breaking change for states that are already integrated, the API does not enforce this rule. States are responsible for enforcing the compact rule themselves.
 #### Example CSV
 ```csv
@@ -110,7 +110,9 @@ If data is not available for a required field, that particular license record ca
 
 Yes. CompactConnect is designed to automatically detect and track changes to license records over time. The Social Security Number (SSN) is the unique identifier CompactConnect uses to create and match individual practitioner accounts. When you upload a license record, CompactConnect uses the provided SSN to determine whether that practitioner already has an account in the system, creating one if not found. If the practitioner's account already exists, CompactConnect will check the differences between the existing license record and the changes uploaded by the state, and apply the changes accordingly.
 
-Because accounts are matched on SSN, simply changing the SSN in your state's system and then uploading the corrected license will **not** update the practitioner's existing CompactConnect account. It will create a brand new, separate account under the new SSN and leave the original account (and any privileges tied to it) unchanged. If a license was previously uploaded with an incorrect SSN, use the `previousSSN` field (see the field table above) when uploading the corrected SSN so CompactConnect can migrate the practitioner's existing account instead of creating a duplicate.
+Because accounts are matched on SSN, simply changing the SSN in your state's system and then uploading the corrected license will **not** update the practitioner's existing CompactConnect account. It will create a brand new, separate account under the new SSN and leave the original account (and any privileges tied to it) unchanged.
+
+**If your state has uploaded a license with an incorrect SSN, contact CSG support before using the `previousSSN` field.** Depending on how far the error has propagated (e.g., whether the practitioner has registered, purchased privileges, or had other actions taken against the license under the incorrect SSN, such as encumbrances or investigations), there may be alternative options for CSG support to help revert or correct the erroneous upload without migrating any accounts. The `previousSSN` field (see the field table above) should be treated as a **last resort**, used only when a state has already uploaded incorrect SSN information that resulted in the practitioner purchasing privileges or other actions being taken against the license under the incorrect SSN, and no alternative remediation is possible. When used, it causes CompactConnect to migrate the practitioner's existing account instead of creating a duplicate.
 
 > **⚠️ Verify SSNs before you upload.** The SSN is the sole identifier CompactConnect uses to match a license to a practitioner's account, and every downstream consequence of an upload (account creation, privilege eligibility, public lookup, etc.) follows from it. Uploading an incorrect SSN is not a low-risk mistake to leave unaddressed, as it silently creates or attaches records to the wrong account, fragmenting the practitioner's licensure history and leaving privileges tied to whichever account was in place at the time they were purchased.
 >

backend/compact-connect/lambdas/nodejs/package.json

@@ -5,7 +5,7 @@
   "description": "NodeJS lambdas for CompactConnect",
   "resolutions": {
     "fast-xml-parser": "5.7.3",
-    "postcss": "8.5.10"
+    "postcss": "8.5.12"
   },
   "scripts": {
     "build": "tsc",

backend/compact-connect/lambdas/nodejs/yarn.lock

@@ -4870,10 +4870,10 @@ pkg-dir@^4.2.0:
   dependencies:
     find-up "^4.0.0"
 
-postcss@8.5.10, postcss@^8.3.11:
-  version "8.5.10"
-  resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.5.10.tgz#8992d8c30acf3f12169e7c09514a12fed7e48356"
-  integrity sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==
+postcss@8.5.12, postcss@^8.3.11:
+  version "8.5.12"
+  resolved "https://registry.yarnpkg.com/postcss/-/postcss-8.5.12.tgz#cd0c0f667f7cb0521e2313234ea6e707a9ec1ddb"
+  integrity sha512-W62t/Se6rA0Az3DfCL0AqJwXuKwBeYg6nOaIgzP+xZ7N5BFCI7DYi1qs6ygUYT6rvfi6t9k65UMLJC+PHZpDAA==
   dependencies:
     nanoid "^3.3.11"
     picocolors "^1.1.1"

backend/compact-connect/lambdas/python/common/cc_common/data_model/data_client.py

@@ -3191,15 +3191,28 @@ def _move_s3_object(self, *, old_key: str, new_key: str):
                 CopySource={'Bucket': self.config.provider_user_bucket_name, 'Key': old_key},
                 Key=new_key,
             )
-            self.config.s3_client.delete_object(Bucket=self.config.provider_user_bucket_name, Key=old_key)
         except ClientError as e:
+            # we intentionally log the keys here so technical support staff
+            # can recover the document if the move fails.
             logger.error(
                 'Failed to move provider document to the new keyspace',
                 old_key=old_key,
                 new_key=new_key,
                 error=str(e),
             )
 
+        try:
+            self.config.s3_client.delete_object(Bucket=self.config.provider_user_bucket_name, Key=old_key)
+            logger.info('Moved provider document to new keyspace')
+        except ClientError as e:
+            # we intentionally log the key here so technical support staff
+            # can complete cleanup of the document if the delete fails.
+            logger.error(
+                'Failed to delete provider document from the old keyspace',
+                old_key=old_key,
+                error=str(e),
+            )
+
     def _build_put_transaction_item(self, record: CCDataClass, condition: dict | None = None) -> dict:
         return {
             'Put': {

backend/compact-connect/lambdas/python/provider-data-v1/handlers/ingest.py

@@ -1,8 +1,9 @@
 import json
 from copy import deepcopy
 
+from aws_lambda_powertools.metrics import MetricUnit
 from boto3.dynamodb.types import TypeSerializer
-from cc_common.config import config, logger
+from cc_common.config import config, logger, metrics
 from cc_common.data_model.provider_record_util import ProviderRecordType, ProviderRecordUtility
 from cc_common.data_model.schema import LicenseRecordSchema
 from cc_common.data_model.schema.common import ActiveInactiveStatus, UpdateCategory
@@ -17,6 +18,14 @@
 license_schema = LicenseIngestSchema()
 license_update_schema = LicenseUpdateRecordSchema()
 
+# Custom metrics tracking how often states rely on the previousSSN last-resort correction feature, split by
+# whether the correction fully migrated the practitioner (old provider had no other licenses), only partially
+# migrated them (other licenses remained on the old provider id), or found nothing to migrate (spurious
+# previousSSN or an already-migrated replay). Each is alarmed on separately in the CDK stack.
+SSN_CORRECTION_FULL_MIGRATION_METRIC = 'ssn-correction-full-migration'
+SSN_CORRECTION_PARTIAL_MIGRATION_METRIC = 'ssn-correction-partial-migration'
+SSN_CORRECTION_NO_MIGRATION_METRIC = 'ssn-correction-no-migration'
+
 
 @sqs_handler
 def preprocess_license_ingest(message: dict):
@@ -387,25 +396,32 @@ def _perform_ssn_correction_migration(
     old Cognito user deletion and re-registration email follow here. A concurrency conflict inside the
     migration raises, letting SQS redeliver the message after the visibility timeout.
     """
-    logger.info('Performing SSN correction migration', previous_provider_id=previous_provider_id)
-
-    result = config.data_client.migrate_provider_for_ssn_correction(
-        compact=compact,
-        previous_provider_id=previous_provider_id,
-        new_provider_id=new_provider_id,
-        jurisdiction=jurisdiction,
-        license_type=license_type,
-        new_ssn_last_four=new_ssn_last_four,
-    )
-    if not result.migration_performed:
-        logger.info('No records to migrate for previous provider id; proceeding with normal ingest')
-        return
+    with logger.append_context_keys(previous_provider_id=previous_provider_id):
+        logger.info('Performing SSN correction migration')
 
-    if result.full_migration and result.old_provider_registered_email is not None:
-        _delete_old_cognito_user_and_send_reregistration_email(
+        result = config.data_client.migrate_provider_for_ssn_correction(
             compact=compact,
-            old_registered_email=result.old_provider_registered_email,
+            previous_provider_id=previous_provider_id,
+            new_provider_id=new_provider_id,
+            jurisdiction=jurisdiction,
+            license_type=license_type,
+            new_ssn_last_four=new_ssn_last_four,
         )
+        if not result.migration_performed:
+            logger.info('No records to migrate for previous provider id; proceeding with normal ingest')
+            metrics.add_metric(name=SSN_CORRECTION_NO_MIGRATION_METRIC, unit=MetricUnit.Count, value=1)
+            return
+
+        if result.full_migration:
+            metrics.add_metric(name=SSN_CORRECTION_FULL_MIGRATION_METRIC, unit=MetricUnit.Count, value=1)
+        else:
+            metrics.add_metric(name=SSN_CORRECTION_PARTIAL_MIGRATION_METRIC, unit=MetricUnit.Count, value=1)
+
+        if result.full_migration and result.old_provider_registered_email is not None:
+            _delete_old_cognito_user_and_send_reregistration_email(
+                compact=compact,
+                old_registered_email=result.old_provider_registered_email,
+            )
 
 
 def _delete_old_cognito_user_and_send_reregistration_email(*, compact: str, old_registered_email: str):
@@ -424,7 +440,10 @@ def _delete_old_cognito_user_and_send_reregistration_email(*, compact: str, old_
         return
 
     logger.info('Deleted old Cognito user after SSN correction; sending re-registration email')
-    config.email_service_client.send_provider_ssn_correction_reregistration_email(
-        compact=compact,
-        provider_email=old_registered_email,
-    )
+    try:
+        config.email_service_client.send_provider_ssn_correction_reregistration_email(
+            compact=compact,
+            provider_email=old_registered_email,
+        )
+    except Exception as e:  # noqa: BLE001 - best-effort notification. Should not cause batch to fail.
+        logger.error('Failed to send re-registration email after SSN correction migration', error=str(e))

backend/compact-connect/lambdas/python/provider-data-v1/handlers/provider_s3_events.py

@@ -17,6 +17,19 @@ def process_provider_s3_events(event: dict, context: LambdaContext):  # noqa: AR
             size = record['s3']['object']['size']
             logger.info('Object', s3_url=f's3://{bucket_name}/{key}', size=size)
 
+            # "ObjectCreated:Copy" events fire when the SSN-correction migration copies a practitioner's
+            # documents from the old provider id's keyspace to the new one (see
+            # DataClient.migrate_provider_for_ssn_correction). The DynamoDB records have already been fully
+            # migrated by that point, so we treat this as a no-op rather than re-processing it as a fresh
+            # upload.
+            event_name = record.get('eventName', '')
+            if event_name.startswith('ObjectCreated:Copy'):
+                logger.info(
+                    'Ignoring S3 object copy event; records already migrated',
+                    event_name=event_name,
+                )
+                continue
+
             # Provider objects are stored under the following keyspace prefix:
             # compact/{compact}/provider/{provider_id}/document-type/military-affiliations/...
             # we split the key to get the various parts needed to query for the record

backend/compact-connect/lambdas/python/provider-data-v1/tests/function/test_handlers/test_ingest.py

@@ -2,6 +2,7 @@
 from datetime import date, datetime
 from unittest.mock import MagicMock, patch
 
+from aws_lambda_powertools.metrics import MetricUnit
 from cc_common.data_model.update_tier_enum import UpdateTierEnum
 from common_test.test_constants import DEFAULT_PROVIDER_ID
 from moto import mock_aws
@@ -1111,7 +1112,32 @@ def test_partial_migration_keeps_cognito_user_and_sends_no_email(self):
         self.assertTrue(self._does_old_cognito_user_exist())
         self._mock_send_reregistration_email.assert_not_called()
 
-    def test_no_op_migration_still_ingests_license_normally(self):
+    @patch('handlers.ingest.metrics')
+    def test_full_migration_emits_full_migration_metric(self, mock_metrics):
+        self._put_old_provider_records()
+        self._create_old_cognito_user()
+
+        resp = self._run_ingest_with_previous_provider_id()
+        self.assertEqual({'batchItemFailures': []}, resp)
+
+        mock_metrics.add_metric.assert_called_once_with(
+            name='ssn-correction-full-migration', unit=MetricUnit.Count, value=1
+        )
+
+    @patch('handlers.ingest.metrics')
+    def test_partial_migration_emits_partial_migration_metric(self, mock_metrics):
+        self._put_old_provider_records(with_second_license=True)
+        self._create_old_cognito_user()
+
+        resp = self._run_ingest_with_previous_provider_id()
+        self.assertEqual({'batchItemFailures': []}, resp)
+
+        mock_metrics.add_metric.assert_called_once_with(
+            name='ssn-correction-partial-migration', unit=MetricUnit.Count, value=1
+        )
+
+    @patch('handlers.ingest.metrics')
+    def test_no_op_migration_still_ingests_license_normally(self, mock_metrics):
         # the previousSSN resolved to a provider id with no records at all
         resp = self._run_ingest_with_previous_provider_id()
         self.assertEqual({'batchItemFailures': []}, resp)
@@ -1126,6 +1152,32 @@ def test_no_op_migration_still_ingests_license_normally(self):
 
         self._mock_send_reregistration_email.assert_not_called()
 
+        # a no-op previousSSN path still emits a metric so we can detect spurious or already-migrated corrections
+        mock_metrics.add_metric.assert_called_once_with(
+            name='ssn-correction-no-migration', unit=MetricUnit.Count, value=1
+        )
+
+    def test_full_migration_still_succeeds_when_reregistration_email_fails(self):
+        """A re-registration email failure must not fail the ingest record: the DynamoDB/S3 migration and the
+        old Cognito user deletion have already completed by the time the email is sent, so retrying this record
+        would only redeliver a message that no longer has anything left to migrate."""
+        from cc_common.exceptions import CCInternalException
+
+        self._mock_send_reregistration_email.side_effect = CCInternalException('email service unavailable')
+        self._put_old_provider_records()
+        self._create_old_cognito_user()
+
+        resp = self._run_ingest_with_previous_provider_id()
+        self.assertEqual({'batchItemFailures': []}, resp)
+
+        self._mock_send_reregistration_email.assert_called_once_with(
+            compact='aslp',
+            provider_email=self.OLD_REGISTERED_EMAIL,
+        )
+        # the migration itself still completed, despite the email failure
+        self.assertFalse(self._does_old_cognito_user_exist())
+        self.assertEqual([], self._get_provider_records(self.OLD_PROVIDER_ID))
+
     def test_full_migration_with_unregistered_old_provider_sends_no_email(self):
         # the old provider never registered: no Cognito user, no registered email on the provider record
         self.test_data_generator.put_default_provider_record_in_provider_table(is_registered=False)