feat(datasource-pylon): write operations (CRUD) (EXT-11)

[christophebrun-forest]

Closes EXT-11 — Story 7 of EXT-4.

What

The datasource was read-only: every column declared is_read_only: true, and create / update / delete fell through to the contract's NotImplementedError. This opens the write half — the endpoints, the payload, and the ids a filter-driven write applies to — through a mechanism the three collection bases share.

File	Role
client/writes.rb	new: one method per Pylon write endpoint, post / patch / delete
collections/writes.rb	new: create / update / delete, the payload builder, ids_for
collections/base_collection.rb	includes the mixin; add_column(writable:); no_search? moves up; id_values deduplicates
collections/fetch_all_collection.rb	add_column(writable:) for the two in-memory collections
issue.rb / account.rb / contact.rb / team.rb / user.rb	the client calls, the directional fields, the one rename
the four schema_definition.rb	writable: true on the columns Pylon accepts
schema/custom_fields_introspector.rb	honours Pylon's is_read_only; carries multi_value
forest_admin_datasource_pylon.rb	UnsupportedWriteError, PartialWriteError, WriteRejectedError

What may be written is the schema, not a second list

is_read_only on the column is the single source of truth the payload builder reads, the way api_filters already is for filtering. A column the schema declares read-only is dropped from the payload — native, foreign key or custom field alike — and nothing here keeps a parallel list of writable names that could drift from what the UI offers.

A column is writable: true when POST or PATCH accepts it in the shape it is read under. That last clause is what leaves several Json columns read-only although the endpoint takes something by that name: external_ids, phone_numbers, channels hold {external_id, label}-style objects, and the write shape the reference documents is not the one the column shows — writing one for the other would replace the account's ids with something Pylon cannot read back. domain and primary_domain stay read-only for a different reason: they are two projections of the domains list, and writing one would leave the other stale, so domains is the writable one. Everything Pylon computes — the issue number, the link, the timestamps, the counters, source, role_name, is_deactivated — stays read-only.

A foreign key is declared writable even though the emitted schema will not carry it. GeneratorField forces isReadOnly on a foreign key whatever the datasource says, so the detail view has one editor per key rather than two. What the flag opens is that editor: the BelongsTo reads its own read-only state off the key column, and the front sends the choice back as the very column named here. account_id, requester_id, assignee_id, team_id on an issue, account_id on a contact.

A custom field is writable when Pylon flags it writable. EXT-10 declared every one read-only for lack of a write endpoint; the definition's own is_read_only flag is now what decides, Pylon setting it on the fields synced from an app or an integration, which its own endpoints refuse. Only an explicit false opens a field: a definition carrying no flag at all — a renamed key, an endpoint that does not return it, a type predating it — is left read-only and reported once per boot, rather than read as editable. Reading the absence the other way would open every custom field of the collection, the synced ones included, and offer an editor whose every save Pylon rejects; this way costs the capability and says so.

Five decisions worth reviewing

A verb Pylon has no endpoint for refuses with a message. There is no POST or DELETE on a user — an agent is invited and deactivated from Pylon itself — and no DELETE on a team. Left alone, the toolkit's NotImplementedError reaches the operator as an unexpected 500 with nothing in it. The three *_record hooks default to refuse_write, so a collection wires only the verbs that exist and the others answer A PylonUser record cannot be created: the Pylon API exposes no endpoint for it. The hook is also what write_endpoint? reads to refuse before the payload is built and the ids are resolved, rather than a second list of supported verbs a collection would have to keep in step with its own hooks: the refusal holds whatever the selection turns out to be, and everything the write would otherwise do on the way there answers with something else — the cap naming a count, a field of the wrong direction naming a field — sending the operator to narrow a selection that was never the problem, or answering a selection matching no record with a success on a collection that cannot perform one. UnsupportedWriteError descends from ValidationError for the same reason UnsupportedOperatorError does: it names something the operator did and can undo, and the message is the only place they learn what.

A field Pylon accepts in one direction only is dropped when it asks for nothing and refused when it does. The Forest schema carries a single read-only flag per column, so both directions offer body_html, state, type, is_disabled, emails; create_only_fields / update_only_fields is what tells them apart at write time. A form resending an untouched field is not an edit, so a field asking for nothing is dropped silently, and a value the operator really changed is refused, naming the field and the direction — answering an edit with a success Pylon did not perform is worse than an error.

What "asks for nothing" means differs by direction, and only a create can tell without reading: Pylon fills a create in with exactly what a blank value asks for, so a blank one is dropped there. On an update the record already holds a value, and that value is the only thing settling whether the patch changes it — an unchecked box is nothing to write over a stored false, and a real edit over a stored true. Blankness alone would drop the second and report an edit Pylon never performed. Two blanks count as the same state, so an unchecked box over a null still costs nothing, and strings compare stripped, so an editor handing back the markup it was given re-indented does not refuse an edit nobody made.

The read this costs is one request on the collections whose endpoint filters id, and one per record on PylonIssue, which reads an id through its own endpoint — charged only on an update whose patch names a field of the wrong direction, and only for those fields. It is charged against the write budget as well: stored_read? declares it before the ids are resolved, so the cap knows what the patch will owe every record it reaches.

Pylon's own refusal reaches the operator. APIError descends from the package's Error, which the agent's ErrorTranslator does not recognise: it keeps the status and answers 'Unexpected error', so the likeliest way a write fails — a required field left out, a value the endpoint does not accept — would arrive as nothing at all. A 4xx is re-raised as WriteRejectedError, a ValidationError whose message the agent surfaces; a 5xx or a dropped connection is not the operator's to act on and stays the APIError it was, carrying its status. PartialWriteError already travelled this way, and it is the write that made it worth it: a read failing costs a page, where a write failing costs the operator the reason their edit did not land.

A filter-driven write resolves its ids exactly or refuses. An id equals / id in filter with no search is answered without a single request — that is what the record detail and the bulk selection of the UI send, and reading them back to learn ids they just named would spend the budget the writes need. Anything else — a scope, a segment, a search, a condition on another column — goes through the collection's own list, so the scope applies and the endpoint filters what it can. filtered_ids reads the ids off a bare leaf or a top-level and and asserts nothing about the rest of the tree: the leftovers travel to list, which answers them the way a read does.

A selection costing more than one pass is refused rather than written halfway. MAX_WRITE_REQUESTS = 20, and it is the budget of a whole pass rather than of its writes: Pylon writes one record per request against a quota of ten to twenty per minute, so a delete that stopped in the middle of the page would look done and would not be — the very failure this datasource refuses elsewhere.

What the cap bounds is therefore derived from what one record of the write costs. requests_per_record_read is zero where the search endpoint filters id, a whole selection travelling in one request whatever its size, and one on PylonIssue, which reads an id through GET /issues/{id}: a named selection there still reaches twenty, one resolved by reading each named id reaches ten, and six when the patch also has a stored value read. The resolution is charged per record only when the filter names ids — any other selection is answered by a single page of the collection's own read, whose cost does not grow with the count, so it keeps the full reach.

The window still asks for one record past the cap, so an overflow is seen rather than guessed from a full page, the same bound foreign_keys_matching puts on a resolved relation condition. max_resolvable_ids follows the same arithmetic and stays clamped to MAX_ID_LOOKUPS: past that fan-out fetch_by_ids truncates with a warning, and a truncated resolution would write to a subset while reporting the whole. The budget is the tighter of the two at today's numbers; the clamp keeps that true if either moves.

Two shapes on the wire

A custom field is written as a list of entries, not a map. {'slug' => …, 'value' => …} per field under custom_fields, values for a multiselect, and a select by the slug of its option — which is what the Enum column advertises, so the round trip is closed with the read side EXT-10 pinned. The introspector now carries multi_value on each entry for that one branch.

A write answers with the record under data, or it is a broken contract. extract_data hands the body back untouched when data is absent, which is what a read wants and a write must not accept: the collection would serialize the envelope into a record with no id. extract_written raises a typed APIError naming the operation and the body instead. Nothing in this file degrades — best_effort exists for the calls whose result enriches a page, where a missing thread costs a column, whereas a write that silently did nothing would tell the operator their edit landed.

Also on the wire: a nil is dropped on a create, Pylon filling in what is left out, and travels on an update, where it is the operator clearing a value. And the id is escaped before being joined to the path, like every read does.

Four deviations from the plan

The custom-field payload is a list, not {slug: {value}}. The scope spelled the map shape; POST /issues and the other write endpoints take custom_fields as an array of {slug, value} / {slug, values} objects. The list is what travels.

MAX_WRITE_REQUESTS was not in the scope. ids_for was, and resolving ids without bounding them is what makes a bulk delete report a success it did not perform, given a write budget of ten to twenty requests per minute. The cap is the smaller half of that budget, and it is spent by the whole pass rather than by its writes alone: counting the writes only, it let a filter-driven update on PylonIssue spend sixty requests where it was allowed twenty — the 429 outliving its retries, the write stopping mid-selection, which is exactly what the cap is there to refuse. The message tells the operator to narrow the selection, and names the reach that applies to theirs.

id_values deduplicates. id in names the records to act on, and the same one named twice is one record: only resolve_ids_by_list deduplicated, so a named selection carrying a duplicate patched one record twice and, on a delete, answered 404 the second time — reported as a partial failure of a delete that fully succeeded. The caps now count records rather than mentions, and a primary-key lookup no longer spends two requests on one id.

no_search? moved from CursorCollection to BaseCollection. ids_for needs it on every collection, not only on the ones read through a cursor. Same body, one level up; no behaviour change.

Verification

647 examples, 0 failures — coverage 1305/1305 lines (100%, threshold 90), branch 95.0%. RuboCop clean over the 55 files of the package.

Client specs cover: the endpoint each of the eleven write methods reaches, the data envelope unwrapped, the two malformed shapes raising, an APIError carrying the status and the request id, an id escaped so it cannot alter the path, and a rate-limited create retried where a gateway error is not.

Collection specs cover: the writable columns posted and the record serialized back; read-only columns, unknown keys and empty values dropped; custom fields written through value and values with the synced one left out, and no custom_fields key when none was set; a field of the wrong direction refused when changed and dropped when it holds the stored value or nothing; an id filter patched without a read first, an in filter reaching every record, a patch of only read-only keys sending nothing, and a filter carrying more than an id resolved through a read; a delete over a filter and a delete matching nothing; the cap refused before spending a request, refused once the resolution reveals the count, and refused on the collection that cannot filter an id; the reach dividing on PylonIssue when the patch has every record read before it is written and holding when it owes them nothing, the resolution of a named selection refused before the first of its reads, and the full reach kept both where the resolution costs one request whatever the count and on the collection whose read does not fan out; the three verbs Pylon has no endpoint for, refused whatever the selection reaches and before a request is spent on resolving it; a 4xx create and a 4xx patch surfaced with the reason Pylon gave where a 5xx stays what it was; an id named twice written once and counted once against the cap; and the per-collection specifics — an account type written as account_type, an account refused is_disabled on a create, a contact patched, a team created and the members of the record it answers with flattened, a team's members replaced, a user's status patched.

The eight collection specs also re-assert their schemas against the new flags, including the columns that stay read-only and why. The introspector specs cover the three states of Pylon's flag: read-only when it says so, writable when it says so, and read-only with a warning when it says nothing.

🤖 Generated with Claude Code

Note

Add CRUD write operations to datasource-pylon collections

• Adds a shared write layer in writes.rb and HTTP routing in writes.rb to support create, update, and delete actions.
• Marks specific columns as writable in collection schemas and enforces field-direction constraints like CREATE_ONLY and UPDATE_ONLY.
• Adds UnsupportedWriteError, PartialWriteError, and WriteRejectedError to report write rejections and partial failures.
• Custom fields now respect the Pylon is_read_only flag and use the values key for multi-value types.
• Risk: retry_policy.rb restricts auto-retries to GET, HEAD, and OPTIONS; POST and DELETE will only retry on 429.

^{Macroscope summarized 44cd3c2.}

[linear-code]

EXT-11

[qltysh]

11 new issues

Tool	Category	Rule	Count
qlty	Structure	Function with many parameters (count = 4): add_column	6
qlty	Structure	Function with high complexity (count = 5): update	4
qlty	Structure	High total complexity (count = 59)	1

[qltysh]

Function with many parameters (count = 4): add_column [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 4): add_column [qlty:function-parameters]

[macroscopeapp]

🟠 High collections/writes.rb:81

ids_for returns duplicate IDs from the direct id in shortcut, so update and delete invoke the write endpoint repeatedly for the same record; a second delete can fail after the first succeeds, and duplicates can also incorrectly trigger the 20-target limit. Deduplicate the extracted IDs before enforcing the cap and returning the shortcut result.

-        ids  = filtered_ids(tree)
+        ids  = filtered_ids(tree)&.uniq

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/forest_admin_datasource_pylon/lib/forest_admin_datasource_pylon/collections/writes.rb around line 81:

`ids_for` returns duplicate IDs from the direct `id in` shortcut, so `update` and `delete` invoke the write endpoint repeatedly for the same record; a second delete can fail after the first succeeds, and duplicates can also incorrectly trigger the 20-target limit. Deduplicate the extracted IDs before enforcing the cap and returning the shortcut result.

[qltysh]

Function with high complexity (count = 5): write_each [qlty:function-complexity]

[qltysh]

Function with many parameters (count = 4): honour_write_direction [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 4): unchanged_fields [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 5): refuse_partial_write [qlty:function-parameters]

[qltysh]

Function with many parameters (count = 4): build_payload [qlty:function-parameters]

[qltysh]

Function with high complexity (count = 5): honour_write_direction [qlty:function-complexity]

[qltysh]

Function with high complexity (count = 5): update [qlty:function-complexity]

[qltysh]

Function with high complexity (count = 6): ids_for [qlty:function-complexity]