fix(codegen): payload optionality comes from @required, in every port (#309)

C# emitted `public required` on EVERY payload property regardless of `@required`;
Kotlin emitted every property non-null for the same reason. A `template.output`
payload is the type an LLM response deserializes into, so neither could represent
a response omitting a field the metadata never marked required. The generated C#
parser's own doc comment says it throws when the JSON is "missing a `required`
property" — the adopter's report, verbatim, from the code.

THE RULE: absent `@required` ⇒ optional. `spec/metamodel/field.json` types the attr
as an optional boolean defaulting to absent. The DECISION is one cross-language
contract; the RENDERING stays idiomatic — C# `T?`, Kotlin `T? = null`, TS `name?:
T`, Python `T | None = None`.

Only C# and Kotlin changed. TS and Python already read it. Java needed nothing:
SpringTypeMapper already emits boxed Integer/Long/Boolean, so every record
component is nullable by construction and absent ⇒ optional already held. Getting
that right meant checking the type mapper rather than trusting "Java records can't
express optional."

C# was contradicting ITSELF, not merely the spec: its FR-010 extractor derives
required-ness from `@required` and treats an absent optional as benign
LOST_OPTIONAL, and `ExtractorGenerator.cs` documents in prose that PayloadCodegen
"does not honor @required". One port, two tiers, opposite predicates.

Kotlin reuses `KotlinGenUtil.isRequiredField` rather than adding a second
predicate. It accepts boolean `true` or the string `"true"`, matching what
`KotlinExtractSchemaEmitter` already accepts, so the payload type and the mapper
populating it cannot disagree — the lockstep Python's `is_field_required` docstring
protects. Python holds the tighter boolean-only threshold on both its tiers, Kotlin
the looser one on both. Whether `@required: "true"` is legal metadata at all is a
LOADER question and is left open rather than half-decided here.

WHY IT SURVIVED FOUR RELEASES: the tests pinned it. C#'s payload suite asserted
`public required` on fixtures where NO field carried `@required`, so it could not
distinguish "reads the attr" from "hardcodes it" — it would have failed under a
correct implementation. A demo test asserted that omitting an unmarked field "fails
to compile" while calling them "required members"; it now declares what it asserts
and gained the arm nobody had, that omitting an OPTIONAL member must compile.
Kotlin's #270 test used non-null as a proxy for "origins don't decide nullability",
which the all-non-null emitter also satisfied; it now carries both directions.

Every flipped assertion was checked against its own fixture rather than
regenerated. That caught a real error mid-fix: `alphaText`/`betaText` in the SHARED
corpus ARE `@required: true`, so blanket-flipping them would have replaced one
wrong golden with another. Which is the finding — the shared xpkg-collision corpus
already carried both arms, and the ports were asserting contradictory output for
one shared model. It is now the payload tier's optionality oracle too.

Adopter-visible in C# and Kotlin: a payload property that was non-null becomes
nullable unless its metadata declares `@required`.

Verified: C# 1592 (Codegen 354, +1 new arm) · Kotlin 315 · Java reactor BUILD
SUCCESS · codegen-ts 1247 · python codegen 415 · gates 10/10.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DhpswkF1NvwxhFWMmdAT15

Author: dmealing

CHANGELOG.md

@@ -352,6 +352,56 @@ command whose whole job is the drift verdict. Pinned by a regression test on the
 which needs no Postgres container and covers the silent half: before the fix its failure
 output *is* the bug, verbatim.
 
+### Fixed — a payload field's optionality now comes from `@required` in every port ([#309](https://github.com/metaobjectsdev/metaobjects/issues/309), NuGet/Maven)
+
+C# emitted `public required` on **every** generated payload property regardless of
+`@required`, and Kotlin emitted every property non-null for the same reason. So a
+`template.output` payload — the type an LLM response is deserialized into — could not
+represent a response that omits a field the metadata never marked required. The generated
+C# parser's own doc comment says it throws when the JSON is *"missing a `required`
+property"*, which is #309 exactly: adopting the generator meant rejecting real responses
+that a hand-written record accepted.
+
+**The rule, now uniform:** a payload field is optional unless it declares `@required`.
+`spec/metamodel/field.json` documents `@required` as an optional boolean defaulting to
+absent, so absent ⇒ optional. The **decision** is one cross-language contract; the
+**rendering** stays idiomatic — C# `T?`, Kotlin `T? = null`, TypeScript `name?: T`, Python
+`T | None = None`.
+
+**Only C# and Kotlin changed.** TypeScript and Python already read `@required`. Java needed
+no change and got none: `SpringTypeMapper` already emits boxed `Integer`/`Long`/`Boolean`,
+so every record component is nullable by construction and absent ⇒ optional already held.
+
+**C# was contradicting itself, not just the spec.** Its own FR-010 extractor derives
+per-field required-ness from `@required` and classifies an absent optional as benign
+`LOST_OPTIONAL`; `ExtractorGenerator.cs` even documents that PayloadCodegen *"does not
+honor @required"*. One port, two tiers, opposite predicates.
+
+**Kotlin uses that port's existing `KotlinGenUtil.isRequiredField`**, which accepts the
+boolean `true` or the string `"true"` — matching what `KotlinExtractSchemaEmitter` already
+accepts, so the payload type and the mapper that populates it cannot disagree. That lockstep
+is the property Python's `is_field_required` docstring protects; Python holds the tighter
+boolean-only threshold on both of its tiers, Kotlin the looser one on both. Whether
+`@required: "true"` should be legal metadata at all is a loader question, left open.
+
+**What made this survivable for four releases: the tests pinned it.** C#'s payload suite
+asserted `public required` on fixtures where **no field carried `@required`**, so it could
+never distinguish "reads the attr" from "hardcodes it" — and one demo test asserted that
+omitting an unmarked field "fails to compile" while its comment called them "required
+members". That test now declares what it asserts, and gained the arm nobody had: omitting an
+**optional** member must compile. Kotlin's `#270` test asserted non-null as a proxy for
+"origins don't decide nullability", which the all-non-null emitter also satisfied; it now
+carries both arms, so a `@required` field with an `origin.first` stays non-null while an
+unmarked sibling with the same origin kind goes nullable.
+
+The shared `template-output-render-conformance/xpkg-collision` corpus turns out to carry
+both arms already — `alphaText`/`betaText` are `@required: true` while `fromAlpha`/`fromBeta`
+are not — so the ports were asserting **contradictory output for one shared model**. It is
+now the payload tier's optionality oracle as well as its collision oracle.
+
+Adopter-visible in C# and Kotlin: a payload property that was non-null becomes nullable
+unless its metadata declares `@required`. Mark the fields you genuinely require.
+
 ### Fixed — a projection could not borrow an entity's alternate key ([#310](https://github.com/metaobjectsdev/metaobjects/issues/310), all four loaders)
 
 A `object.projection` borrows its key rather than declaring one:

server/csharp/MetaObjects.Codegen.Tests/DemoTests.cs

@@ -28,8 +28,8 @@ public class DemoTests
             { "field.string": { "name": "title" } }
           ]}},
           { "object.projection": { "name": "AuthorBrief", "children": [
-            { "field.string": { "name": "displayName" } },
-            { "field.int": { "name": "postCount" } },
+            { "field.string": { "name": "displayName", "@required": true } },
+            { "field.int": { "name": "postCount", "@required": true } },
             { "field.object": { "name": "posts", "isArray": true, "@objectRef": "PostBrief",
               "children": [ { "origin.collection": { "@via": "Author.posts" } } ] } }
           ]}},
@@ -98,10 +98,15 @@ public static string Go(IProvider p)
     }
 
     [Fact]
-    public void Compile_time__a_wrong_shaped_caller_fails_to_compile()
+    public void Compile_time__a_caller_omitting_a_DECLARED_required_member_fails_to_compile()
     {
-        // The caller omits required members (postCount, posts) — the codegen'd shape
-        // contract makes this a compile error, not a silent runtime mismatch.
+        // The caller omits `postCount`, which the metadata marks `@required: true` — the
+        // codegen'd shape contract makes that a compile error, not a silent runtime mismatch.
+        //
+        // #309: this test previously proved the same thing about fields carrying NO
+        // `@required` at all, because the emitter marked every property `required`. It
+        // therefore pinned the defect while its comment claimed to demonstrate the design.
+        // The fixture now declares what the test asserts, so it passes for the stated reason.
         var source = GeneratedSource(Load()) + """
         public static class BadCaller
         {
@@ -113,7 +118,28 @@ public static string Go(IProvider p)
         }
         """;
         var errors = CompileErrors(source);
-        Assert.True(errors.Count > 0, "expected a wrong-shaped caller to FAIL compilation, but it compiled clean");
+        Assert.True(errors.Count > 0, "expected a caller omitting a required member to FAIL compilation, but it compiled clean");
+    }
+
+    [Fact]
+    public void Compile_time__a_caller_omitting_an_OPTIONAL_member_compiles()
+    {
+        // The other arm, which no test covered while every property was `required`: `posts`
+        // carries no `@required`, so omitting it must be legal. This is the shape #309 was
+        // filed about — an LLM response that simply does not populate an optional field.
+        var source = GeneratedSource(Load()) + """
+        public static class PartialCaller
+        {
+            public static string Go(IProvider p)
+            {
+                var partial = new AuthorBrief { displayName = "Ada", postCount = 1 };
+                return RenderHandles.RenderContentStrategyPrompt(partial, p);
+            }
+        }
+        """;
+        var errors = CompileErrors(source);
+        Assert.True(errors.Count == 0, "expected omitting an OPTIONAL member to compile, got: "
+            + string.Join("; ", errors));
     }
 
     [Fact]

server/csharp/MetaObjects.Codegen.Tests/PayloadCodegenTests.cs

@@ -48,11 +48,11 @@ public void Emits_payload_record_with_scalar_and_nested_array_fields()
     {
         var src = PayloadCodegen.GeneratePayloadRecords(Load(), "AuthorBrief");
         Assert.Contains("public sealed record AuthorBrief", src);
-        Assert.Contains("public required string displayName { get; init; }", src);
-        Assert.Contains("public required int postCount { get; init; }", src);
-        Assert.Contains("public required IReadOnlyList<PostBrief> posts { get; init; }", src);
+        Assert.Contains("public string? displayName { get; init; }", src);
+        Assert.Contains("public int? postCount { get; init; }", src);
+        Assert.Contains("public IReadOnlyList<PostBrief>? posts { get; init; }", src);
         Assert.Contains("public sealed record PostBrief", src);
-        Assert.Contains("public required string title { get; init; }", src);
+        Assert.Contains("public string? title { get; init; }", src);
     }
 
     // Same shape as Model, but the nested @objectRef is authored FULLY-QUALIFIED
@@ -82,7 +82,7 @@ public void Fully_qualified_objectRef_strips_to_bare_record_type_and_resolves_ne
         var root = new MetaDataLoader().Load([new InMemoryStringSource(FqnRefModel, id: "fqn.json")]).Root;
         var src = PayloadCodegen.GeneratePayloadRecords(root, "AuthorBrief");
         // Regression: the FQN must NOT leak into the generated C# type or record name.
-        Assert.Contains("public required IReadOnlyList<PostBrief> posts { get; init; }", src);
+        Assert.Contains("public IReadOnlyList<PostBrief>? posts { get; init; }", src);
         Assert.Contains("public sealed record PostBrief", src);   // nested record DID resolve (FindObject matched the bare name)
         Assert.DoesNotContain("acme::ai::", src);
     }
@@ -188,13 +188,13 @@ public void No_churn_a_non_colliding_closure_emits_the_exact_pre_fix_bare_name_o
         Assert.Equal(
             "public sealed record AuthorBrief\n" +
             "{\n" +
-            "    public required string displayName { get; init; }\n" +
-            "    public required int postCount { get; init; }\n" +
-            "    public required IReadOnlyList<PostBrief> posts { get; init; }\n" +
+            "    public string? displayName { get; init; }\n" +
+            "    public int? postCount { get; init; }\n" +
+            "    public IReadOnlyList<PostBrief>? posts { get; init; }\n" +
             "}\n\n" +
             "public sealed record PostBrief\n" +
             "{\n" +
-            "    public required string title { get; init; }\n" +
+            "    public string? title { get; init; }\n" +
             "}\n",
             src);
     }
@@ -245,16 +245,16 @@ public void Emits_two_distinct_package_qualified_records_not_one_merged_first_wi
         Assert.Equal(
             "public sealed record Digest\n" +
             "{\n" +
-            "    public required AcmeAlphaNote fromAlpha { get; init; }\n" +
-            "    public required AcmeBetaNote fromBeta { get; init; }\n" +
+            "    public AcmeAlphaNote? fromAlpha { get; init; }\n" +
+            "    public AcmeBetaNote? fromBeta { get; init; }\n" +
             "}\n\n" +
             "public sealed record AcmeAlphaNote\n" +
             "{\n" +
-            "    public required string alphaText { get; init; }\n" +
+            "    public string? alphaText { get; init; }\n" +
             "}\n\n" +
             "public sealed record AcmeBetaNote\n" +
             "{\n" +
-            "    public required string betaText { get; init; }\n" +
+            "    public string? betaText { get; init; }\n" +
             "}\n",
             src);
         Assert.DoesNotContain("record Note", src);

server/csharp/MetaObjects.Codegen.Tests/PayloadGeneratorTests.cs

@@ -83,11 +83,11 @@ public void Emits_full_record_for_bare_payloadRef_in_a_named_package()
         var file = Assert.Single(files);
         Assert.Equal("ClassificationResponse.payload.cs", file.Path);
         Assert.Contains("public sealed record ClassificationResponse", file.Content);
-        Assert.Contains("public required string documentType { get; init; }", file.Content);
-        Assert.Contains("public required double confidence { get; init; }", file.Content);
-        Assert.Contains("public required NoteEntry note { get; init; }", file.Content);
+        Assert.Contains("public string? documentType { get; init; }", file.Content);
+        Assert.Contains("public double? confidence { get; init; }", file.Content);
+        Assert.Contains("public NoteEntry? note { get; init; }", file.Content);
         Assert.Contains("public sealed record NoteEntry", file.Content);
-        Assert.Contains("public required string value { get; init; }", file.Content);
+        Assert.Contains("public string? value { get; init; }", file.Content);
         // The FQN must not leak into the file name or the record/type names.
         Assert.DoesNotContain("acme::intake::", file.Content);
         Assert.DoesNotContain("::", file.Path);
@@ -149,11 +149,11 @@ public void Collision_within_one_payloadRefs_closure_emits_two_distinct_qualifie
         Assert.Equal("Digest.payload.cs", file.Path);
         Assert.Contains("public sealed record Digest", file.Content);
         Assert.Contains("public sealed record AcmeAlphaNote", file.Content);
-        Assert.Contains("public required string alphaText { get; init; }", file.Content);
+        Assert.Contains("public string? alphaText { get; init; }", file.Content);
         Assert.Contains("public sealed record AcmeBetaNote", file.Content);
-        Assert.Contains("public required string betaText { get; init; }", file.Content);
-        Assert.Contains("public required AcmeAlphaNote fromAlpha { get; init; }", file.Content);
-        Assert.Contains("public required AcmeBetaNote fromBeta { get; init; }", file.Content);
+        Assert.Contains("public string? betaText { get; init; }", file.Content);
+        Assert.Contains("public AcmeAlphaNote? fromAlpha { get; init; }", file.Content);
+        Assert.Contains("public AcmeBetaNote? fromBeta { get; init; }", file.Content);
         Assert.DoesNotContain("public sealed record Note", file.Content);
     }
 
@@ -202,14 +202,14 @@ public void Origin_children_are_ignored_for_typing_declared_type_wins()
         var file = Assert.Single(files);
         Assert.Equal("Digest.payload.cs", file.Path);
         // Declared `field.int` wins over the (`@convert`-acknowledged) string passthrough.
-        Assert.Contains("public required int alias { get; init; }", file.Content);
+        Assert.Contains("public int? alias { get; init; }", file.Content);
         Assert.DoesNotContain("string alias", file.Content);
         // Declared `field.string` wins over origin.collection — no list, no via-target type.
-        Assert.Contains("public required string summary { get; init; }", file.Content);
+        Assert.Contains("public string? summary { get; init; }", file.Content);
         // Declared `field.object @objectRef` + isArray wins over the disagreeing @via walk.
-        Assert.Contains("public required IReadOnlyList<Highlight> posts { get; init; }", file.Content);
+        Assert.Contains("public IReadOnlyList<Highlight>? posts { get; init; }", file.Content);
         Assert.Contains("public sealed record Highlight", file.Content);
-        Assert.Contains("public required string snippet { get; init; }", file.Content);
+        Assert.Contains("public string? snippet { get; init; }", file.Content);
         // The ignored @via entity never enters the closure.
         Assert.DoesNotContain("record Post", file.Content);
         Assert.DoesNotContain("internalNotes", file.Content);

server/csharp/MetaObjects.Codegen.Tests/RenderHelperConformanceTests.cs

@@ -245,12 +245,17 @@ public void Document_DigestDoc_resolves_fqn_nested_objectRef_across_collision()
         // merged `{ alphaText; betaText }` shape.
         Assert.Contains("public sealed record AcmeAlphaNote", records);
         Assert.Contains("public sealed record AcmeBetaNote", records);
+        // #309 — this fixture carries BOTH arms, which is what makes it the payload tier's
+        // optionality oracle as well as its collision oracle: the shared corpus declares
+        // `alphaText`/`betaText` as `@required: true` (meta.alpha.json / meta.beta.json)
+        // while `fromAlpha`/`fromBeta` carry no `@required` (meta.app.json). A port that
+        // hardcodes either answer now fails on the other half of the same model.
         Assert.Contains("public required string alphaText { get; init; }", records);
         Assert.Contains("public required string betaText { get; init; }", records);
         Assert.DoesNotContain("public sealed record Note", records);
         // Digest's own fields point at the qualified names, not at each other's field.
-        Assert.Contains("public required AcmeAlphaNote fromAlpha { get; init; }", records);
-        Assert.Contains("public required AcmeBetaNote fromBeta { get; init; }", records);
+        Assert.Contains("public AcmeAlphaNote? fromAlpha { get; init; }", records);
+        Assert.Contains("public AcmeBetaNote? fromBeta { get; init; }", records);
 
         var payloadSrc = "namespace Acme.Generated;\n" + records;
         var asm = CompileToAssembly(file.Content, payloadSrc);

server/csharp/MetaObjects.Codegen/PayloadCodegen.cs

@@ -332,7 +332,22 @@ private static void EmitClosureRecords(
                     // Scalar array (e.g. field.string with isArray) -> a list of the scalar.
                     type = IsArrayField(f) ? $"IReadOnlyList<{scalar}>" : scalar;
                 }
-                lines.Add($"    public required {type} {f.Name} {{ get; init; }}");
+                // #309 — optionality comes from the DECLARED @required, like every other
+                // tier. `required` on an unmarked field made the generated
+                // `JsonSerializer.Deserialize<Payload>` reject any response omitting it,
+                // which is fatal for the payload's main job: absorbing an LLM response that
+                // does not always populate every field. It also contradicted this port's own
+                // sibling — the FR-010 extractor derives per-field required-ness from
+                // `@required` and classifies an absent optional as benign LOST_OPTIONAL.
+                //
+                // `spec/metamodel/field.json` documents @required as an optional boolean
+                // defaulting to absent, so absent ⇒ optional. Rendered idiomatically per
+                // language: C# uses `required` + non-nullable for a required field and a
+                // nullable property with no `required` otherwise.
+                bool isRequired = f.Attr(FIELD_ATTR_REQUIRED) is true;
+                lines.Add(isRequired
+                    ? $"    public required {type} {f.Name} {{ get; init; }}"
+                    : $"    public {type}? {f.Name} {{ get; init; }}");
             }
             lines.Add("}");
             output.Add(string.Join("\n", lines));