feat: add Cloudflare Queues destination#907
Draft
alexluong wants to merge 6 commits into
Draft
Conversation
Add support for publishing events to Cloudflare Queues via their HTTP API. Implements #655 - Add destcfqueues Go provider with Publish, Validate, and ComputeTarget - Add provider metadata (metadata.json and instructions.md) - Update OpenAPI spec with CloudflareQueues schemas - Add user documentation (cloudflare-queues.mdx) - Register provider in default.go - Add validation and publish unit tests - Add SDK test factory and integration tests
Resolves conflicts in: - docs/apis/openapi.yaml: kept both Kafka (#779) and Cloudflare Queues destinations; added created_at/updated_at/disabled_at fields and x-docs-type per post-#895 conventions; added merge-patch null-tolerant delivery_metadata/metadata to Update schema - internal/destregistry/providers/default.go: kept both Kafka and Cloudflare Queues provider registrations - docs: replaced docs/pages/destinations/cloudflare-queues.mdx with docs/content/destinations/cloudflare-queues.mdoc to match the docs/content/ + markdoc reorganization on main; added to nav.json, overview.mdoc, concepts.mdoc - destcfqueues.go: switched from BaseProvider.MakeHTTPClient to package-level destregistry.NewHTTPClient
The original implementation posted a {"messages":[{"body":...}]} batch
wrapper to the single-message push endpoint, which is the wrong shape
for that path, and parsed `result` as []map (the docs show an object).
On a real success response the slice/object mismatch would silently
fall into the `apiResponse.Success` zero-value path and report every
successful publish as failed.
Verified against:
- developers.cloudflare.com/api/resources/queues/subresources/messages/methods/push/
Changes:
- Format(): drop the messages array wrapper; post
{"body":{"data":..., "metadata":...}, "content_type":"json"} directly
to /accounts/{id}/queues/{id}/messages
- cloudflareAPIResponse.Result is now an object matching CF docs
(result.metadata.metrics.{backlog_bytes,backlog_count,oldest_message_timestamp_ms})
- cloudflareAPIError gains documentation_url and source.pointer
- Publish(): non-2xx always returns failure with as much detail as
parseable; 2xx with unparseable body now trusts the status instead
of falling through to the success=false branch
- Tests rewritten to mock the real CF success shape; added a roundtrip
guard that fails loudly if CF's documented shape stops decoding
- Added nav.json + overview/concepts mdoc entries for the destination
- Replace SetHTTPClient + testTransport URL-rewriting hack with a
WithBaseURL functional option on New(). Tests now point baseURL at
the httptest.Server directly — no path rewriting required.
- Dashboard URL: the per-queue dashboard path uses the queue *name*,
not the queue *id* we store in config, so the previous
/queues/{queue_id} link 404s. Point at the account's queues list
page (/workers/queues) instead until we surface the queue name too.
- OpenAPI: use realistic 32-char hex examples for account_id/queue_id, align target_url example with the new dashboard-list-page format - metadata.json: clarify that account_id/queue_id are hex IDs, not names - instructions.md: replace the camelCase 'contentType' note with the actual API payload shape (body wrapper + content_type:json) and link to the correct CF docs page
The SDK contract tests can't run until SDKs are regenerated from the updated OpenAPI, and Kafka shipped without one (#779). Defer this file + factory entry to the follow-up SDK regen PR where the generated SDK shape will be known.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Cloudflare Queues as a destination type — events are published via Cloudflare's HTTP API to
POST /accounts/{account_id}/queues/{queue_id}/messages, one event per request.Continuation of #657 (originally authored by @Swahjak).
Closes #655. Supersedes #657.
Summary
destcfqueuesprovider implementingValidate,CreatePublisher,ComputeTarget{"body":{"data":...,"metadata":...},"content_type":"json"}WithBaseURLtest option for injecting anhttptest.Server(no transport-rewriting hack)DestinationCloudflareQueues, Create, Update) wired into all discriminator maps;DestinationTypeenum extendeddocs/content/destinations/cloudflare-queues.mdocplus nav/overview/concepts entriesTest plan
go build ./...go vet ./...go test ./internal/destregistry/providers/destcfqueues/...(12 tests)swagger-cli validate docs/apis/openapi.yamlsuccessResponseJSONfixture with a recorded real response at the same time.spec-sdk-testsupdate — deferred to a follow-up PR post-merge.On mocks
Tests use
httptest.Serverbut the success fixture is derived from CF's documented response shape.TestCloudflareAPIResponse_DecodesDocumentedShapeis a roundtrip guard: if CF's documented shape ever stops decoding cleanly into our types, it fails loudly.