Define dataflows for ecosystems-review-owned integrations - #3100
Conversation
Split of #3093 by CODEOWNERS team. Jira: TXP-277
philjlee
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1a7a854677
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - id: anecdote-logs | ||
| always_on: true | ||
| granular: false | ||
| data_type: logs | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here: anecdote/README.md:7 states that every newly reported bug produces a Datadog event, and anecdote/manifest.json:42-45 classifies metrics, logs, and events as submitted data, but this file exposes only the first two. Consumers of this metadata will therefore omit a supported Anecdote signal.
Useful? React with 👍 / 👎.
| - id: buoyant-cloud-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here. The setup in buoyant_cloud/README.md:17-22 lets users send events or metrics and validates both in their respective explorers, while manifest.json sets creates_events to true; declaring only metrics makes the new dataflow metadata incomplete whenever a notification rule sends rollout or health events.
Useful? React with 👍 / 👎.
| - id: census-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here. census/README.md:23-37 documents that sync completion events and metrics are transmitted to Datadog, and census/manifest.json:24-25 marks both as submitted data, so dataflow consumers currently see only half of the integration's telemetry.
Useful? React with 👍 / 👎.
| - id: cloudsmith-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here. cloudsmith/README.md:9-12 explicitly maps security, audit, policy, member, and quota data to Datadog events, and the check calls self.event for those records, so the metrics-only declaration causes product dataflow metadata to hide a substantial supported signal.
Useful? React with 👍 / 👎.
| - id: gatling-enterprise-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here. gatling_enterprise/manifest.json:42-43 marks both metrics and events as submitted, and the bundled dashboard queries injection-start and injection-end records with data_source: events; the current declaration therefore omits the event signal used by the integration's own dashboard.
Useful? React with 👍 / 👎.
| - id: stormforge-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here. stormforge/README.md:34-38 explicitly says that application updates and applied recommendations create events, so dataflow consumers will receive an incomplete view when those events are emitted even though the new file claims metrics are the integration's only inbound signal.
Useful? React with 👍 / 👎.
| - id: tidb-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound logs entry here. TiDB explicitly collects TiDB, TiKV, TiFlash, and slow-query logs (tidb/README.md:5-8), provides log-collection configuration starting at tidb/README.md:67, and declares assets.logs.source as tidb; when users enable that documented configuration, the metrics-only dataflow omits their logs.
Useful? React with 👍 / 👎.
| - id: ziris-logs | ||
| always_on: true | ||
| granular: false | ||
| data_type: logs | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound traces entry here. watchtower_ziris/README.md:7 says the integration collects metrics, logs, and traces over OpenTelemetry, its validation instructs users to verify that spans are streaming (README.md:53), and the manifest classifies traces as submitted data; the new metadata currently drops that third signal entirely.
Useful? React with 👍 / 👎.
| - id: hasura-cloud-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add inbound logs and traces entries here. hasura_cloud/README.md:8-9 says the integration exports operation logs, metrics, and traces, and README.md:17-26 instructs users to configure log facets before confirming that all three signals are automatically sent when a project receives traffic; the metrics-only declaration hides two supported signals.
Useful? React with 👍 / 👎.
| - id: zebrium-metrics | ||
| always_on: true | ||
| granular: false | ||
| data_type: metrics | ||
| direction: inbound |
There was a problem hiding this comment.
Add an inbound events entry here. zebrium/README.md:5-9 repeatedly states that Zebrium detection events and metrics are sent directly to Datadog, and the bundled dashboard overlays those detections as events; declaring only metrics omits the signal that carries each root-cause detection.
Useful? React with 👍 / 👎.
1 participant
Jira: TXP-277
Adds
assets/dataflows.yamlto 63 integrations owned by@DataDog/ecosystems-review, declaring 70 dataflows. Mechanical change: no code, no metrics, no manifests touched.Precedent: #2925 "Define dataflows for saas-integrations", which created 13 of the 14 existing dataflow files in this repo.
Selection criteria
A directory is in this batch only if all of the following hold:
assets/dataflows.yamltoday.manifest.json. The dataflows validator hard-requires one —dataflows_validation_handler.go:56.data_typeis mechanically derivable from a committed artifact, with no judgement call:metadata.csvwith at least one data row →metricsassets/logs/*.yaml→logs.github/CODEOWNERSresolves<dir>/assets/dataflows.yaml(last-match-wins) to@DataDog/ecosystems-review.Every directory in this batch has
@DataDog/ecosystems-reviewas its Datadog team CODEOWNER, so review from that team covers the whole PR. External co-owners are auto-requested on many of these paths, but their approval is not a gate.Field values
always_on: true/granular: false/direction: inboundmatches 14 of the 17 dataflow entries already in the repo. The only deviation in the repo isvercel, which isalways_on: falsefor a Serverless product-enablement reason that does not apply here.Dataflow IDs are
<app_id>-<data_type>, takingapp_idfrommanifest.jsonrather than the directory name — e.g.doppler/→doppler-logs,zilliz_cloud_zilliz_cloud/→zilliz-cloud-zilliz-cloud-metrics. All IDs were checked for collisions against every existing ID in the repo, and against each other.Validation
These files were validated by executing the real validator,
DataflowsValidationHandlerfromdd-source/domains/integrationscatalog/libs/catalogassetslib/dataflows_validation_handler.go, atddoghq/dd-source@main, against everydataflows.yamlfile in the repo working tree with this PR applied. Result: 0 failures, covering per-file unmarshalling, proto constraint validation, and the cross-fileHandleLibraryID-uniqueness check.The harness was negative-tested first and confirmed to reject: a missing
always_on; adata_typeoutsidevalidDataTypes; anidbreaking^[a-z0-9-]+$; anidunder 3 characters; a file with neitherprovidesnoruses; a.ymlextension; a missingmanifest.json; and the same dataflow ID provided by two apps.This matters because APW does not post validator comments on integrations-extras PRs (
enable_validator_commentsis set only forpub-platform-stagingandpublishing-platform). A malformeddataflows.yamlmerges cleanly here and only fails afterwards, in the shared asset pipeline.No
data_typewas guessed anywhere. Any integration whose correct value was not unambiguous is deferred to a later batch rather than approximated.Contents
Metrics only (45)
1eone-e-metricsablyably-metricsagora_analyticsagora-analytics-metricsakeyless_gatewayakeyless-gateway-metricsalgorithmiaalgorithmia-metricsauthzed_cloudauthzed-cloud-metricsbuoyant_cloudbuoyant-cloud-metricscensuscensus-metricscloudsmithcloudsmith-metricscribl_streamcribl-stream-metricscyralcyral-metricsembrace_mobileembrace-mobile-metricsemqxemqx-metricsfiddlerfiddler-metricsgatling_enterprisegatling-enterprise-metricshasura_cloudhasura-cloud-metricsinngestinngest-metricsn2wsn2ws-metricsneo4jneo4j-metricsneoloadneoload-metricsnn_sdwannn-sdwan-metricsns1ns1-metricsoceanbasecloudoceanbase-cloud-metricspacketfabricpacketfabric-metricsperfectscaleperfectscale-metricspostmanpostman-metricspurefapurefa-metricspurefbpurefb-metricsredisenterpriseredisenterprise-metricsscalrscalr-metricsscaphandrescaphandre-metricssinglestoredb_cloudsinglestoredb-cloud-metricssofy_sofysofy-metricssosiviososivio-metricsstarburst_galaxystarburst-galaxy-metricsstatsigstatsig-metricsstormforgestormforge-metricssuperwisesuperwise-metricstidbtidb-metricstidb_cloudtidb-cloud-metricsunitqunitq-metricsupbound_uxpupbound-uxp-metricsupstashupstash-metricszebriumzebrium-metricszilliz_cloud_zilliz_cloudzilliz-cloud-zilliz-cloud-metricsLogs only (11)
appomniappomni-logscontrast_security_adrcontrast-security-adr-logsdagsterdagster-plus-logsdopplerdoppler-logsfaunafauna-logsinvaryinvary-logsjfrog_platform_cloudjfrog-platform-cloud-logslambdatestlambdatest-logsngrokngrok-logstwingatetwingate-logstypingdna_activelocktypingdna-activelock-logsMetrics and logs (7)
aimonaimon-metrics,aimon-logsanecdoteanecdote-metrics,anecdote-logsbottomline_recordandreplaybottomline-recordandreplay-metrics,bottomline-recordandreplay-logscelerdatacelerdata-metrics,celerdata-logsgraviteegravitee-metrics,gravitee-logsjfrog_platform_self_hostedjfrog-platform-metrics,jfrog-platform-logswatchtower_zirisziris-metrics,ziris-logs🤖 Generated with Claude Code