Recorded Future Identity - HTTP Data Collector Deprecation - #14944
Open
Erik Mangsten (ErikMangstenRecFut) wants to merge 8 commits into
Open
Recorded Future Identity - HTTP Data Collector Deprecation#14944Erik Mangsten (ErikMangstenRecFut) wants to merge 8 commits into
Erik Mangsten (ErikMangstenRecFut) wants to merge 8 commits into
Conversation
They are anyway hardcoded within DCE/DCR, no need for params
Erik Mangsten (ErikMangstenRecFut)
requested review from
a team
as code owners
August 19, 2026 11:41
Erik Mangsten (ErikMangstenRecFut)
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR updates the legacy Recorded Future Identity v3.0 playbooks to migrate off the deprecated Azure Log Analytics Data Collector connector and instead ingest logs via the Azure Monitor Logs Ingestion API (DCE/DCR), including updated documentation and deployment templates.
Changes:
- Add new shared Data Connectors infrastructure ARM template (DCE + 3 DCRs + 3 Log Analytics tables + connector tile).
- Update v3.0 playbook ARM templates to use Managed Identity + HTTP ingestion to DCE/DCR and auto-assign required RBAC roles.
- Refresh v3.0 README with migration notes, new table names, and updated deployment steps/parameters.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| Solutions/Recorded Future Identity/Playbooks/v3.0/readme.md | Documents migration deadline, new deployment steps, updated table names/permissions, and refreshes deploy links. |
| Solutions/Recorded Future Identity/Playbooks/v3.0/RFI-search-workforce-user/azuredeploy.json | Migrates ingestion to HTTP + Managed Identity, adds DCE/DCR parameters, and adds optional RBAC auto-assignment. |
| Solutions/Recorded Future Identity/Playbooks/v3.0/RFI-search-external-user/azuredeploy.json | Migrates ingestion to HTTP + Managed Identity, adds DCE/DCR parameters, and adds optional RBAC auto-assignment. |
| Solutions/Recorded Future Identity/Playbooks/v3.0/RFI-lookup-and-save-user/azuredeploy.json | Migrates ingestion to HTTP + Managed Identity, adds DCE/DCR parameters, and adds optional RBAC auto-assignment. |
| Solutions/Recorded Future Identity/Data Connectors/azuredeploy-v3.json | New ARM template to deploy shared DCE/DCR/table infrastructure and the Sentinel connector tile for v3.0 playbooks. |
| Solutions/Recorded Future Identity/Data Connectors/azuredeploy-alert-importer.json | Adds a sync note indicating the DCE definition must match the new v3 template. |
| Solutions/Recorded Future Identity/Data Connectors/README.md | Documents the new azuredeploy-v3.json and clarifies the shared DCE duplication constraints. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+118
to
+119
| <a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Frecordedfuture%2FAzure-Sentinel%2Frfpd-77178-log-ingestion-api-2%2FSolutions%2FRecorded%20Future%20Identity%2FData%20Connectors%2Fazuredeploy-v3.json" target="_blank"></a> | ||
| <a href="https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Frecordedfuture%2FAzure-Sentinel%2Frfpd-77178-log-ingestion-api-2%2FSolutions%2FRecorded%20Future%20Identity%2FData%20Connectors%2Fazuredeploy-v3.json" target="_blank"></a> |
Comment on lines
+7
to
+12
| "prerequisites": [ | ||
| "The custom connector RFI-CustomConnector-0-1-0 have to be deployed under the same subscription.", | ||
| "To use the Recorded Future for Azure connector, you will need a valid API token from Recorded Future as described in the [documentation](https://learn.microsoft.com/en-us/connectors/recordedfuturev2/#how-to-get-credentials)", | ||
| "The v3.0 Data Connectors infrastructure (azuredeploy-v3.json) must be deployed before this playbook.", | ||
| "The following playbooks must be deployed before this playbook: RFI-add-EntraID-security-group-user, RFI-confirm-EntraID-risky-user, RFI-lookup-and-save-user" | ||
| ], |
Comment on lines
+612
to
624
| "type": "Http", | ||
| "inputs": { | ||
| "body": "@{outputs('Transform_new_Credential_dump_exposures_array_into_a_JSON_object')}", | ||
| "method": "POST", | ||
| "uri": "@{parameters('DceEndpoint')}/dataCollectionRules/@{parameters('DcrCredentialDumpsImmutableId')}/streams/@{parameters('StreamCredentialDumpsName')}?api-version=2023-01-01", | ||
| "headers": { | ||
| "Log-Type": "@parameters('credential_dumps_log_analytics_custom_log_name')", | ||
| "time-generated-field": "@{utcNow()}" | ||
| "Content-Type": "application/json" | ||
| }, | ||
| "host": { | ||
| "connection": { | ||
| "name": "@parameters('$connections')['azureloganalyticsdatacollector']['connectionId']" | ||
| } | ||
| }, | ||
| "method": "post", | ||
| "path": "/api/logs" | ||
| "body": "@outputs('Transform_new_Credential_dump_exposures_array_into_a_JSON_object')", | ||
| "authentication": { | ||
| "type": "ManagedServiceIdentity", | ||
| "audience": "https://monitor.azure.com" | ||
| } | ||
| } |
Comment on lines
156
to
164
| "data": { | ||
| "lookup_lookback_date": "@formatDateTime(addDays(utcNow(), if(equals(triggerBody()?['lookup_lookback_days'], null), parameters('lookup_lookback_days_default'), triggerBody()?['lookup_lookback_days'])), 'yyyy-MM-dd')", | ||
| "lookup_results": "@body('Credential_Lookup_V2_-_Look_up_credential_data_for_one_or_more_users')", | ||
| "lookup_results_log_analytics_custom_log_name": "@if(equals(triggerBody()?['lookup_results_log_analytics_custom_log_name'], null), parameters('lookup_results_log_analytics_custom_log_name_default'), triggerBody()?['lookup_results_log_analytics_custom_log_name'])", | ||
| "lookup_results_log_analytics_custom_log_name": "RFI_UsersLookupResults_V2_CL", | ||
| "parameters_passed": { | ||
| "lookup_lookback_days": "@triggerBody()?['lookup_lookback_days']", | ||
| "lookup_results_log_analytics_custom_log_name": "@triggerBody()?['lookup_results_log_analytics_custom_log_name']", | ||
| "risky_user_email": "@triggerBody()?['risky_user_email']" | ||
| } | ||
| }, |
Comment on lines
+5
to
+8
| > [!IMPORTANT] | ||
| > ### Log Ingestion API migration (deadline: 2026-09-14) | ||
| > | ||
| > These playbooks previously used the deprecated Azure Log Analytics Data Collector connector, which Microsoft is retiring on September 14, 2026. They have been updated to use the Log Ingestion API via a Data Collection Endpoint (DCE) and Data Collection Rules (DCRs). |
Contributor
|
Hi Erik Mangsten (@ErikMangstenRecFut) , Please provide screenshots of the playbook workflow showing that it has executed successfully. Additionally, provide screenshots of the new table demonstrating that data is being ingested into the table after the playbook changes were implemented. Once the validation is complete, please package the solution using the V3 tool https://github.com/Azure/Azure-Sentinel/blob/master/Tools/Create-Azure-Sentinel-Solution/V3/README.md and commit the changes. After committing, review any Copilot-generated review comments, address the feedback, and resolve all associated review conversations. |
Contributor
Author
|
v-rusraut As I explicitly stated, this is not part of a content hub package and therefor does not need packaging. I will check the Copilot comments. wrt. screenshots of the workflow execution, is this a new requirement? Have never required that before during my years maintaining this solution, and this has been part of our solution for a number of years, this is just a maintenance change due to HTTP Data Collector, which we have multiple PRs and merged code... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change(s):
Reason for Change(s):
NOTE
This is not part of a specific package, rather this is purely a "GitHub" release, so we can support and help our customers that still use this fucntionality.