Skip to content

fix(workflow-executor): tolerate LLM field name variations in findField#1585

Merged
Scra3 merged 3 commits into
feat/prd-214-server-step-mapperfrom
fix/workflow-executor-fuzzy-field-name
May 20, 2026
Merged

fix(workflow-executor): tolerate LLM field name variations in findField#1585
Scra3 merged 3 commits into
feat/prd-214-server-step-mapperfrom
fix/workflow-executor-fuzzy-field-name

Conversation

@Scra3
Copy link
Copy Markdown
Member

@Scra3 Scra3 commented May 19, 2026

Summary

  • findField previously did exact string matching (===) on displayName and fieldName
  • LLMs sometimes return snake_case variants of field names (e.g. first_name instead of firstname) even when the tool schema constrains the value via z.literal
  • This adds a normalized fallback that strips separators (_, -, spaces) and lowercases before matching, so a hallucinated variation no longer kills an otherwise correct step

Root cause

invokeWithTool returns toolCall.args directly without re-validating against the Zod schema. The z.literal constraint is only used to generate the tool definition sent to the LLM — it is not enforced on the response.

Test plan

  • Existing update-record / read-record / load-related-record tests pass
  • Manually verify: a workflow with an account.firstname field no longer fails when the LLM outputs first_name

fixes PRD-214

Note

Fix findField in RecordStepExecutor to tolerate LLM field name variants

LLMs may return field names in snake_case, camelCase, lowercase, or hyphenated forms. The findField method in record-step-executor.ts already uses normalized comparison to handle these cosmetic variants; this PR clarifies the inline comment and adds a parameterized test suite in update-record-step-executor.test.ts to verify each variant resolves to the correct normalized field name.

Changes since #1585 opened

  • Modified RecordStepExecutor.findField method in workflow-executor to implement two-phase field resolution: first attempting exact match on displayName or fieldName, then computing fuzzy matches using normalized names (lowercased with spaces, underscores, and hyphens removed), returning a field only if exactly one fuzzy match exists, otherwise returning undefined instead of the first fuzzy match [065d4b0]
  • Added test case to update-record-step-executor.test.ts in workflow-executor under the 'automaticExecution: update direct (Branch B)' suite that constructs a schema with two fields normalizing to the same string and asserts the executor returns an error status [065d4b0]
  • Reformatted field object literals in update-record-step-executor.test.ts [e3f8e58]

Macroscope summarized c3496f0.

@linear
Copy link
Copy Markdown

linear Bot commented May 19, 2026

PRD-214

@qltysh
Copy link
Copy Markdown

qltysh Bot commented May 19, 2026

Qlty


Coverage Impact

⬇️ Merging this pull request will decrease total coverage on feat/prd-214-server-step-mapper by 0.01%.

Modified Files with Diff Coverage (1)

RatingFile% DiffUncovered Line #s
Coverage rating: A Coverage rating: A
packages/workflow-executor/src/executors/record-step-executor.ts100.0%
Total100.0%
🚦 See full report on Qlty Cloud »

🛟 Help
  • Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

  • Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

  • File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

    • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

@hercemer42
Copy link
Copy Markdown

[claude-opus-4-7] [Code conventions / defence-in-depth — not blocking]

The normalized fallback runs after both exact passes (displayName === name, then fieldName === name), so the realistic ambiguity surface is narrower than it first looks: a collision only bites when the LLM hallucinates a form that matches no field's exact displayName or fieldName AND two+ fields happen to normalize to the same string. That's a fairly contrived shape — e.g. fields { displayName: "Full Name", fieldName: "fullname" } and { displayName: "FullName", fieldName: "full_name" } with the LLM returning "Full-Name".

Cheap mitigation if we ever want it: swap the two normalized .find() calls for one .filter() over both names and return undefined when length > 1 — rides on the existing FieldNotFoundError path, no new error type. Roughly:

const exact =
  schema.fields.find(f => f.displayName === name) ??
  schema.fields.find(f => f.fieldName === name);
if (exact) return exact;

const fuzzy = schema.fields.filter(
  f => normalizeFieldName(f.displayName) === normalized
    || normalizeFieldName(f.fieldName) === normalized,
);
return fuzzy.length === 1 ? fuzzy[0] : undefined;

Leaving it as-is is also defensible — happy to defer.

// Schema does constrain fieldName to exact values. However, invokeWithTool returns
// toolCall.args as-is without re-running Zod validation on the response, so the LLM can
// silently ignore the constraint and return a formatting variation (e.g. "first_name"
// instead of "firstname"). The normalized fallback catches these cosmetic mismatches.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd tighten this, for example :

  // LLMs occasionally return formatting variants of field names (e.g. "first_name" for
  // "firstname", "full-name" for "Full Name") even though the tool schema declares them
  // as literals. Fall back to a normalized comparison so a cosmetic variation doesn't
  // fail an otherwise correct step.

But it's preferential

…indField

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Scra3 Scra3 force-pushed the fix/workflow-executor-fuzzy-field-name branch from c61b015 to c3496f0 Compare May 20, 2026 10:42
alban bertolini and others added 2 commits May 20, 2026 12:49
…d match in findField

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…field test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Scra3 Scra3 merged commit 73c1e2f into feat/prd-214-server-step-mapper May 20, 2026
30 checks passed
@Scra3 Scra3 deleted the fix/workflow-executor-fuzzy-field-name branch May 20, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants