diff --git a/docs/guide/java/extra-fields.md b/docs/guide/java/extra-fields.md new file mode 100644 index 0000000000..8982345188 --- /dev/null +++ b/docs/guide/java/extra-fields.md @@ -0,0 +1,188 @@ +--- +title: Extra Fields +sidebar_position: 7 +id: extra_fields +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +This page describes `ForyExtraFields` [Java], an opt-in mechanism for capturing and preserving fields +that exist in a writer's schema but are unknown to the reader's class. + +## The Problem + +In [compatible mode](schema-evolution.md#compatible-mode), a reader can deserialize a payload whose +writer schema differs from its own. When the writer includes a field that the reader's class does +not declare, that field is **silently dropped**. + +## The Solution + +A class opts in by declaring **a field** of type `ForyExtraFields`: + +```java +import org.apache.fory.serializer.ForyExtraFields; + +public class PersonView { + public int age; // known field + public ForyExtraFields extraFields; // sink for everything else +} +``` + +When Fory deserializes a payload into `PersonView`: + +- Fields that match the local class (`age`) are read normally. +- Every unmatched remote field is **captured** into `extraFields` instead of being discarded. +- If no fields are unmatched, `extraFields` stays `null` — the sink is only allocated on demand. + +On re-serialization, Fory emits the **original (remote) schema** and replays every field — matched +fields from the object, unmatched fields from the sink — so a downstream peer with the full schema +recovers them exactly. + +The field can be named anything; detection is by type, not name, and Fory walks the class hierarchy, +so the sink may also be inherited from a superclass. + +A class uses a **single** sink. All of an object's unmatched fields are captured into one +name-keyed map, so a second `ForyExtraFields` field would have no distinct role to play — Fory uses +the first one it finds. This is a limit **per class, not per object +graph**: every object in a nested structure resolves its own sink from its own class, so a parent and +its nested children can each declare one and capture independently — see +[Nested and Collection Fields](#nested-and-collection-fields). + +## Requirements + +- **Compatible mode.** Extra-field capture depends on schema metadata, so it works only when + compatible mode (the default). + [same-schema mode](schema-evolution.md#same-schema-optimization). +- **A sink field** of type `ForyExtraFields` per class. +- Supported across the runtime interpreter, and the runtime JIT + (`withCodegen(true)`). + Cross-language (xlang) mode is not supported. Using `ForyExtraFields` under xlang mode will cause errors. + +## Constraints + +Replay needs a Fory that has deserialized a payload carrying that schema and still +has it cached, ie the instance that produced this object. + +## Reading Captured Fields + +`ForyExtraFields` exposes read-only, name-keyed lookup of the captured values. The sink is populated by +Fory during deserialization; application code reads from it by field name: + +```java +byte[] bytes = /* payload written by a full-schema peer */; +PersonView view = fory.deserialize(bytes, PersonView.class); + +if (view.extraFields != null) { + // Field values are keyed by their original field name. + String name = (String) view.extraFields.get("name"); + + // getOrDefault avoids a null value for absent keys. + int score = (Integer) view.extraFields.getOrDefault("score", 0); + + // Presence check — distinguishes "captured as null" from "not captured". + boolean hasName = view.extraFields.containsKey("name"); + + // Print the captured values. + System.out.println(name); + System.out.println(score); +} +``` + +| Method | Description | +| ------------------------- | ----------------------------------------------------------------- | +| `get(name)` | Captured value for `name`, or `null` if absent (or captured null) | +| `getOrDefault(name, def)` | Captured value, or `def` if the key is absent | +| `containsKey(name)` | Whether a value was captured for `name` (true even if it is null) | +| `isEmpty()` | Whether no fields were captured | +| `getTypeDef()` | The remote schema (`TypeDef`) the fields were captured from | + +Notes: + +- **Primitives are boxed.** A captured `int` reads back as an `Integer`, a `long` as a `Long`, and + so on, because the map holds `Object` values. +- **`null` values are preserved.** A field that was written as `null` is captured as a `null` entry + (not omitted), so `containsKey` returns `true` while `get` returns `null`. On re-serialization it + is replayed as `null`. +- The map is **read-only** to application code; there is no public way to add or mutate captured + entries. + +## Forwarding: Round-Trip Recovery + +When a partial-schema peer re-serializes an object +whose sink is populated, Fory writes the object under the **original remote schema** so the next peer +sees byte layout identical to what the original writer would have produced: + +```java +// --- A: full-schema writer --- +Fory foryA = Fory.builder().withXlang(false).withCompatible(true).build(); +Upstream original = new Upstream(/* f0..f9 */); +byte[] aToB = foryA.serialize(original); + +// --- B: partial-schema intermediary with a sink --- +Fory foryB = Fory.builder().withXlang(false).withCompatible(true).build(); +DownstreamWithSink b = foryB.deserialize(aToB, DownstreamWithSink.class); +// b knows f0..f8; f9 was captured into b.extraFields. + +byte[] bToC = foryB.serialize(b); // replays the full A-schema, including f9 + +// --- C: full-schema receiver recovers everything --- +Fory foryC = Fory.builder().withXlang(false).withCompatible(true).build(); +Upstream recovered = foryC.deserialize(bToC, Upstream.class); +// recovered.f9 equals original.f9 +``` + +This works across **multi-hop chains**. Each hop matches the fields it knows and +captures the rest, and every hop re-stamps the original schema, so a peer several hops downstream +still recovers the full object. + +### Reference Sharing + +If the original writer had two fields aliasing the same object, that identity is preserved through +capture and replay + +```java +DownstreamRefSink result = fory.deserialize(bytes, DownstreamRefSink.class); +Object shared = result.extraFields.get("shared"); +Object alias = result.extraFields.get("alias"); +assert shared == alias; // same instance, identity preserved +``` + +### Nested and Collection Fields + +Capture works with Fory's existing type system. Unknown fields may themselves be objects, collections, arrays, or nested structures; they are captured and replayed using Fory's normal serialization mechanisms. + +The "one sink per class" limit is per class rather than per object graph, **each object in a +nested structure captures into its own sink**, resolved from its own class. A parent with no sink can +hold a child that has one, siblings can each capture independently, and this works to any depth: + +```java +// Full graph: Outer → Middle → Inner{f0, f1} +// Partial graph the reader knows: Outer → Middle → Inner{f0, ForyExtraFields} +OuterPartial b = fory.deserialize(bytes, OuterPartial.class); + +// The unknown Inner.f1 lands in the innermost object's own sink: +Object f1 = b.middle.inner.extraFields.get("f1"); +``` + +Each sink preserves the remote schema of the object it belongs to, so re-serializing the whole graph +replays every level correctly. This is exercised directly in `ForyExtraFieldsTest` — see +`roundTripTripleNestedObjectWithInnerSink` (three levels deep), and +`roundTripMultipleNestedObjectsWithOwnSinks` (two sibling children capturing independently). + +## Future Work + +Adiding Extra-field capture for +the annotation processor, Scala macros, and KSP is planned for a follow-up change. diff --git a/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java b/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java index 7bdda7394c..28bbfa9716 100644 --- a/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java +++ b/java/fory-core/src/main/java/org/apache/fory/builder/BaseObjectCodecBuilder.java @@ -133,6 +133,7 @@ import org.apache.fory.serializer.DeferedLazySerializer.DeferredLazyObjectSerializer; import org.apache.fory.serializer.EnumSerializer; import org.apache.fory.serializer.FinalFieldReplaceResolveSerializer; +import org.apache.fory.serializer.ForyExtraFields; import org.apache.fory.serializer.ObjectSerializer; import org.apache.fory.serializer.PrimitiveSerializers.LongSerializer; import org.apache.fory.serializer.ReplaceResolveSerializer; @@ -922,15 +923,28 @@ protected Expression writeForNotNullNonFinalObject( classInfo, inlineInvoke(typeResolverRef, "getTypeInfo", classInfoTypeRef, clsExpr)))); } - writeClassAndObject.add( + // Mirror WriteContext.writeRef + Expression extraFieldsReplay = + new Expression.LogicalAnd( + inlineInvoke(classInfo, "hasExtraFieldsSink", PRIMITIVE_BOOLEAN_TYPE), + new Invoke( + writeContextRef(), + "tryWriteExtraFieldsSchema", + PRIMITIVE_BOOLEAN_TYPE, + typeResolverRef, + classInfo, + inputObject)); + ListExpression normalWrite = new ListExpression(); + normalWrite.add( typeResolver(r -> r.writeClassExpr(typeResolverRef, writeContextRef(), classInfo))); - writeClassAndObject.add( + normalWrite.add( new Invoke( invokeInline(classInfo, "getSerializer", getSerializerType(clz)), writeMethodName, PRIMITIVE_VOID_TYPE, writeContextRef(), inputObject)); + writeClassAndObject.add(new If(not(extraFieldsReplay), normalWrite)); Expression write = exactClassWrite == null ? writeClassAndObject @@ -945,11 +959,26 @@ private Expression exactClassWrite(Expression inputObject, Expression buffer, Cl } Reference typeInfo = addExactTypeInfoField(clz); Expression serializer = getOrCreateSerializer(clz); - return new ListExpression( - typeResolver( - r -> r.writeExactClassExpr(typeResolverRef, writeContextRef(), buffer, typeInfo, clz)), + ListExpression normalWrite = + new ListExpression( + typeResolver( + r -> + r.writeExactClassExpr( + typeResolverRef, writeContextRef(), buffer, typeInfo, clz)), + new Invoke( + serializer, writeMethodName, PRIMITIVE_VOID_TYPE, writeContextRef(), inputObject)); + if (ForyExtraFields.findSinkField(clz) == null) { + return normalWrite; + } + Expression extraFieldsReplay = new Invoke( - serializer, writeMethodName, PRIMITIVE_VOID_TYPE, writeContextRef(), inputObject)); + writeContextRef(), + "tryWriteExtraFieldsSchema", + PRIMITIVE_BOOLEAN_TYPE, + typeResolverRef, + typeInfo, + inputObject); + return new If(not(extraFieldsReplay), normalWrite); } protected Expression writeTypeInfo( diff --git a/java/fory-core/src/main/java/org/apache/fory/builder/CompatibleCodecBuilder.java b/java/fory-core/src/main/java/org/apache/fory/builder/CompatibleCodecBuilder.java index 691b7715dc..e9671c1f3b 100644 --- a/java/fory-core/src/main/java/org/apache/fory/builder/CompatibleCodecBuilder.java +++ b/java/fory-core/src/main/java/org/apache/fory/builder/CompatibleCodecBuilder.java @@ -39,20 +39,25 @@ import org.apache.fory.builder.Generated.GeneratedCompatibleSerializer; import org.apache.fory.codegen.CodeGenerator; import org.apache.fory.codegen.Expression; +import org.apache.fory.codegen.Expression.Invoke; import org.apache.fory.codegen.Expression.Literal; import org.apache.fory.codegen.Expression.StaticInvoke; import org.apache.fory.config.ForyBuilder; import org.apache.fory.context.ReadContext; +import org.apache.fory.context.WriteContext; import org.apache.fory.logging.Logger; import org.apache.fory.logging.LoggerFactory; import org.apache.fory.memory.MemoryBuffer; import org.apache.fory.meta.TypeDef; import org.apache.fory.platform.GraalvmSupport; +import org.apache.fory.reflect.FieldAccessor; +import org.apache.fory.reflect.ReflectionUtils; import org.apache.fory.reflect.TypeRef; import org.apache.fory.resolver.TypeResolver; import org.apache.fory.serializer.CodegenSerializer; import org.apache.fory.serializer.CompatibleSerializer; import org.apache.fory.serializer.FieldGroups.SerializationFieldInfo; +import org.apache.fory.serializer.ForyExtraFields; import org.apache.fory.serializer.ObjectSerializer; import org.apache.fory.serializer.Serializer; import org.apache.fory.serializer.Serializers; @@ -93,13 +98,20 @@ public class CompatibleCodecBuilder extends ObjectCodecBuilder { private static final String REMOTE_FIELD_INFOS_NAME = "_f_remoteFieldInfos"; private static final String LOCAL_FIELD_INFOS_NAME = "_f_localFieldInfosByRemoteOrder"; private static final String CONSTRUCTOR_TYPE_DEF_NAME = "_f_typeDef"; + private static final String EXTRA_FIELDS_SINK_ACCESSOR_NAME = "_f_extraFieldsSinkAccessor"; + private static final String EXTRA_FIELDS_LOCAL_NAME = "_f_extra"; private final TypeDef typeDef; private final String defaultValueLanguage; private final DefaultValueUtils.DefaultValueField[] defaultValueFields; private final Map fieldInfoIds = new IdentityHashMap<>(); + private final Field extraFieldsSinkField; private String remoteFieldInfosName; private String localFieldInfosName; + private String extraFieldsSinkAccessorName; + private String storedTypeDefFieldName; + private Expression extraFieldsSinkExprBean; + private Expression extraFieldsSinkExpr; public CompatibleCodecBuilder(TypeRef beanType, Fory fory, TypeDef typeDef) { super(beanType, fory, GeneratedCompatibleSerializer.class); @@ -153,6 +165,7 @@ public CompatibleCodecBuilder(TypeRef beanType, Fory fory, TypeDef typeDef) { } this.defaultValueLanguage = defaultValueLanguage; this.defaultValueFields = defaultValueFields; + this.extraFieldsSinkField = ForyExtraFields.findSinkField(beanClass); } // Must be static to be shared across the whole process life. @@ -225,6 +238,16 @@ public String genCode() { decodeCode); ctx.overrideMethod( readMethodName, decodeCode, Object.class, ReadContext.class, READ_CONTEXT_NAME); + if (extraFieldsSinkField != null) { + ctx.overrideMethod( + writeMethodName, + buildExtraFieldsWriteCode(), + void.class, + WriteContext.class, + WRITE_CONTEXT_NAME, + Object.class, + ROOT_OBJECT_NAME); + } registerJITNotifyCallback(); ctx.addConstructor( constructorCode, @@ -263,9 +286,62 @@ public static Serializer setCodegenSerializer( return Serializers.newSerializer(typeResolver, cls, serializerClass); } - @Override - public Expression buildEncodeExpression() { - throw new IllegalStateException("unreachable"); + /** + * Generates the body of the {@code write} method for classes that declare a {@link + * ForyExtraFields} sink. Mirrors {@link CompatibleSerializer#write}. + */ + private String buildExtraFieldsWriteCode() { + extraFieldsSinkAccessorExpr(); + ctx.clearExprState(); + String encodeCode = buildEncodeExpression().genCode(ctx).code(); + encodeCode = ctx.optimizeMethodCode(encodeCode); + encodeCode = encodeCode == null ? "" : encodeCode; + StringBuilder body = new StringBuilder(); + appendBufferAndRefWriterDeclarations(body, encodeCode); + body.append( + StringUtils.format( + "${extraType} ${extra} = (${extraType}) ${accessor}.getObject(${obj});\n" + + "if (${extra} == null || ${extra}.isEmpty()) {\n" + + " this.${serializer}.write(${writeContext}, ${obj});\n" + + " return;\n" + + "}\n", + "extraType", + ctx.type(ForyExtraFields.class), + "extra", + EXTRA_FIELDS_LOCAL_NAME, + "accessor", + extraFieldsSinkAccessorName, + "obj", + ROOT_OBJECT_NAME, + "serializer", + SERIALIZER_FIELD_NAME, + "writeContext", + WRITE_CONTEXT_NAME)); + body.append(encodeCode); + return body.toString(); + } + + private void appendBufferAndRefWriterDeclarations(StringBuilder body, String encodeCode) { + body.append( + StringUtils.format( + "${bufferType} ${buffer} = ${writeContext}.getBuffer();\n", + "bufferType", + ctx.type(MemoryBuffer.class), + "buffer", + BUFFER_NAME, + "writeContext", + WRITE_CONTEXT_NAME)); + if (encodeCode.contains(REF_WRITER_NAME)) { + body.append( + StringUtils.format( + "${refWriterType} ${refWriter} = (${refWriterType}) ${writeContext}.getRefWriter();\n", + "refWriterType", + ctx.type(concreteRefWriterType), + "refWriter", + REF_WRITER_NAME, + "writeContext", + WRITE_CONTEXT_NAME)); + } } @Override @@ -373,7 +449,8 @@ protected Expression deserializeGroupForRecord( } private boolean shouldSkipField(Descriptor descriptor) { - return descriptor.getField() == null + return extraFieldsSinkField == null + && descriptor.getField() == null && descriptor.getFieldConverter() == null && !descriptor.getRawType().isPrimitive(); } @@ -386,6 +463,24 @@ private Expression skipField(Descriptor descriptor) { remoteFieldInfo(descriptor)); } + @Override + protected Expression getFieldValue(Expression bean, Descriptor descriptor) { + if (extraFieldsSinkField != null && descriptor.getField() == null) { + if (extraFieldsSinkExpr == null || extraFieldsSinkExprBean != bean) { + extraFieldsSinkExprBean = bean; + extraFieldsSinkExpr = + tryInlineCast( + new Invoke(extraFieldsSinkAccessorExpr(), "getObject", OBJECT_TYPE, bean), + TypeRef.of(ForyExtraFields.class)); + } + Invoke raw = + new Invoke( + extraFieldsSinkExpr, "get", OBJECT_TYPE, Literal.ofString(descriptor.getName())); + return tryInlineCast(raw, descriptor.getTypeRef()); + } + return super.getFieldValue(bean, descriptor); + } + @Override protected Expression setFieldValue(Expression bean, Descriptor descriptor, Expression value) { if (descriptor.getField() == null) { @@ -393,15 +488,68 @@ protected Expression setFieldValue(Expression bean, Descriptor descriptor, Expre if (converter != null) { return setFieldConverterTargetValue(bean, descriptor, converter, value); } - // Field doesn't exist in current class, skip set this field value. + // If Field doesn't exist in current class, + if (extraFieldsSinkField != null) { + return new StaticInvoke( + ForyExtraFields.class, + "capture", + TypeRef.of(void.class), + bean, + extraFieldsSinkAccessorExpr(), + typeDefFieldExpr(), + Literal.ofString(descriptor.getName()), + value); + } // Note that the field value shouldn't be an inlined value, otherwise field value read may - // be ignored. - // Add an ignored call here to make expression type to void. + // be ignored. Add an ignored call here to make expression type to void. return new StaticInvoke(ExceptionUtils.class, "ignore", value); } return super.setFieldValue(bean, descriptor, value); } + private String ensureLazyFieldGenerated( + String cachedFieldName, String nameConstant, TypeRef fieldType, Expression initializer) { + if (cachedFieldName == null) { + cachedFieldName = ctx.newName(nameConstant); + ctx.addField(false, true, ctx.type(fieldType), cachedFieldName, initializer); + } + return cachedFieldName; + } + + private Expression extraFieldsSinkAccessorExpr() { + extraFieldsSinkAccessorName = + ensureLazyFieldGenerated( + extraFieldsSinkAccessorName, + EXTRA_FIELDS_SINK_ACCESSOR_NAME, + TypeRef.of(FieldAccessor.class), + new StaticInvoke( + FieldAccessor.class, + "createAccessor", + TypeRef.of(FieldAccessor.class), + getOrCreateField( + true, + Field.class, + extraFieldsSinkField.getName() + "SinkField", + () -> + new StaticInvoke( + ReflectionUtils.class, + "getField", + TypeRef.of(Field.class), + staticBeanClassExpr(), + Literal.ofString(extraFieldsSinkField.getName()))))); + return new Expression.Reference(extraFieldsSinkAccessorName, TypeRef.of(FieldAccessor.class)); + } + + private Expression typeDefFieldExpr() { + storedTypeDefFieldName = + ensureLazyFieldGenerated( + storedTypeDefFieldName, + "_f_storedTypeDef", + TypeRef.of(TypeDef.class), + new Expression.Reference(CONSTRUCTOR_TYPE_DEF_NAME, TypeRef.of(TypeDef.class))); + return new Expression.Reference(storedTypeDefFieldName, TypeRef.of(TypeDef.class)); + } + private Expression setFieldConverterTargetValue( Expression bean, Descriptor descriptor, FieldConverter converter, Expression value) { Field field = converter.getField(); diff --git a/java/fory-core/src/main/java/org/apache/fory/context/WriteContext.java b/java/fory-core/src/main/java/org/apache/fory/context/WriteContext.java index bf74c0a67a..5c07dcc8f9 100644 --- a/java/fory-core/src/main/java/org/apache/fory/context/WriteContext.java +++ b/java/fory-core/src/main/java/org/apache/fory/context/WriteContext.java @@ -24,12 +24,14 @@ import org.apache.fory.config.Config; import org.apache.fory.config.Int64Encoding; import org.apache.fory.memory.MemoryBuffer; +import org.apache.fory.reflect.FieldAccessor; import org.apache.fory.resolver.ClassResolver; import org.apache.fory.resolver.TypeInfo; import org.apache.fory.resolver.TypeInfoHolder; import org.apache.fory.resolver.TypeResolver; import org.apache.fory.serializer.BufferCallback; import org.apache.fory.serializer.BufferObject; +import org.apache.fory.serializer.ForyExtraFields; import org.apache.fory.serializer.PrimitiveSerializers.LongSerializer; import org.apache.fory.serializer.Serializer; import org.apache.fory.serializer.StringSerializer; @@ -442,6 +444,58 @@ public BufferCallback getBufferCallback() { return bufferCallback; } + /** + * If {@code obj} carries a populated {@link ForyExtraFields} sink whose remote {@code TypeDef} + * differs from the local type, emits that remote schema header and replays every field (matched + * fields from the bean, unmatched fields from the sink) so a downstream peer with the full schema + * can recover them, excluded from writeRef(obj,typeinfo) because it is only called by methods + * which pass collection objects, and from writeNonRef(obj,typeinfo) which is gated on + * useDeclaredTypeInfo and !crossLanguage ie. the path for mononmorphic elements + */ + public boolean tryWriteExtraFieldsSchema(TypeResolver resolver, TypeInfo typeInfo, Object obj) { + FieldAccessor sinkAccessor = typeInfo.getExtraFieldsSinkAccessor(); + if (sinkAccessor == null) { + return false; + } + ForyExtraFields extraField = (ForyExtraFields) sinkAccessor.getObject(obj); + if (extraField == null || extraField.getTypeDef() == null) { + return false; + } + long sinkTypeDefId = extraField.getTypeDef().getId(); + // The object was deserialized under a different (fuller) remote schema. Emit that remote schema + // header (from the writer-class TypeInfo) and replay all fields through the reader-class + // serializer that captured them, whose field order follows the remote schema. + TypeInfo headerTypeInfo = resolver.getTypeInfoByTypeDefId(sinkTypeDefId); + if (headerTypeInfo == null) { + throw new IllegalStateException( + "Cannot re-serialize " + + obj.getClass().getName() + + " with captured extra fields: this Fory instance does not have the object's original " + + "remote schema (TypeDef id " + + sinkTypeDefId + + ") cached, so it cannot emit the schema header needed to preserve those fields. " + + "Replay needs a Fory that has deserialized a payload carrying that schema and still " + + "has it cached, ie the instance that produced this object. Each instance " + + "retains only a bounded number of distinct remote schemas, so under very high schema " + + "churn the schema may not be retained and the captured fields cannot be replayed."); + } + Serializer replaySerializer = + resolver.getExtraFieldsWriteSerializer(obj.getClass(), sinkTypeDefId); + if (replaySerializer == null) { + throw new IllegalStateException( + "Cannot replay captured extra fields on " + + obj.getClass() + + ": remote TypeDef " + + sinkTypeDefId + + " has not been read into this class on this Fory instance."); + } + resolver.writeTypeInfo(this, headerTypeInfo); + depth++; + ((Serializer) replaySerializer).write(this, obj); + depth--; + return true; + } + /** * Writes a nullable reference-tracked object together with any required type metadata. * @@ -459,6 +513,9 @@ public void writeRef(Object obj) { depth--; return; } + if (typeInfo.hasExtraFieldsSink() && tryWriteExtraFieldsSchema(resolver, typeInfo, obj)) { + return; + } resolver.writeTypeInfo(this, typeInfo); writeData(typeInfo, obj); } @@ -476,6 +533,9 @@ public void writeRef(Object obj, TypeInfoHolder classInfoHolder) { depth--; return; } + if (typeInfo.hasExtraFieldsSink() && tryWriteExtraFieldsSchema(resolver, typeInfo, obj)) { + return; + } resolver.writeTypeInfo(this, typeInfo); writeData(typeInfo, obj); } @@ -543,16 +603,7 @@ public void writeRootRef(Object obj) { return; } buffer.writeByte(Fory.NOT_NULL_VALUE_FLAG); - TypeResolver resolver = typeResolver; - TypeInfo typeInfo = resolver.getTypeInfo(obj.getClass(), rootTypeInfoHolder); - if (crossLanguage && typeInfo.getType() == UnknownStruct.class) { - depth++; - typeInfo.getSerializer().write(this, obj); - depth--; - return; - } - resolver.writeTypeInfo(this, typeInfo); - writeData(typeInfo, obj); + writeNonRef(obj, rootTypeInfoHolder); } /** @@ -570,6 +621,9 @@ public void writeNonRef(Object obj) { depth--; return; } + if (typeInfo.hasExtraFieldsSink() && tryWriteExtraFieldsSchema(resolver, typeInfo, obj)) { + return; + } resolver.writeTypeInfo(this, typeInfo); writeData(typeInfo, obj); } @@ -591,6 +645,9 @@ public void writeNonRef(Object obj, TypeInfoHolder holder) { depth--; return; } + if (typeInfo.hasExtraFieldsSink() && tryWriteExtraFieldsSchema(resolver, typeInfo, obj)) { + return; + } resolver.writeTypeInfo(this, typeInfo); writeData(typeInfo, obj); } diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/TypeInfo.java b/java/fory-core/src/main/java/org/apache/fory/resolver/TypeInfo.java index 862c24c3a8..ab700e9c86 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/TypeInfo.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/TypeInfo.java @@ -26,7 +26,9 @@ import org.apache.fory.meta.EncodedMetaString; import org.apache.fory.meta.Encoders; import org.apache.fory.meta.TypeDef; +import org.apache.fory.reflect.FieldAccessor; import org.apache.fory.reflect.ReflectionUtils; +import org.apache.fory.serializer.ForyExtraFields; import org.apache.fory.serializer.Serializer; import org.apache.fory.type.Types; import org.apache.fory.util.function.Functions; @@ -48,6 +50,8 @@ public class TypeInfo { Serializer serializer; TypeDef typeDef; boolean needToWriteTypeDef; + // Non-null only when this type declares a ForyExtraFields sink. Resolved once when the serializer + FieldAccessor extraFieldsSinkAccessor; TypeInfo( Class type, @@ -170,6 +174,15 @@ public void setSerializer(Serializer serializer) { void setSerializer(TypeResolver resolver, Serializer serializer) { this.serializer = serializer; needToWriteTypeDef = serializer != null && resolver.needToWriteTypeDef(serializer); + this.extraFieldsSinkAccessor = type == null ? null : ForyExtraFields.findSinkAccessor(type); + } + + public FieldAccessor getExtraFieldsSinkAccessor() { + return extraFieldsSinkAccessor; + } + + public boolean hasExtraFieldsSink() { + return extraFieldsSinkAccessor != null; } public String decodeNamespace() { diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java b/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java index 352de334ad..ef3bd3cbe8 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java @@ -83,6 +83,7 @@ import org.apache.fory.serializer.CodegenSerializer; import org.apache.fory.serializer.CodegenSerializer.LazyInitBeanSerializer; import org.apache.fory.serializer.CompatibleSerializer; +import org.apache.fory.serializer.ForyExtraFields; import org.apache.fory.serializer.ObjectSerializer; import org.apache.fory.serializer.PrimitiveSerializers; import org.apache.fory.serializer.Serializer; @@ -514,6 +515,30 @@ public boolean isMapDescriptor(Descriptor descriptor) { public abstract TypeInfo getTypeInfo(Class cls, TypeInfoHolder classInfoHolder); + public final TypeInfo getTypeInfoByTypeDefId(long typeDefId) { + return extRegistry.typeInfoByTypeDefId.get(typeDefId); + } + + public final Serializer getExtraFieldsWriteSerializer(Class cls, long typeDefId) { + Map> idToSerializer = extRegistry.extraFieldsSerializers.get(cls); + return idToSerializer != null ? idToSerializer.get(typeDefId) : null; + } + + private static boolean hasExtraFieldsSinkField(Class cls) { + return ForyExtraFields.findSinkField(cls) != null; + } + + /** + * Records the reader-class serializer used to replay {@code cls} under the remote {@code + * typeDefId} it was captured from. + */ + private void registerExtraFieldsSerializer(Class cls, long typeDefId, Serializer gen) { + extRegistry + .extraFieldsSerializers + .computeIfAbsent(cls, k -> new ConcurrentHashMap<>()) + .putIfAbsent(typeDefId, gen); + } + /** * Writes class info to buffer using the unified type system. This is the single implementation * shared by both ClassResolver and XtypeResolver. @@ -1283,9 +1308,13 @@ private TypeInfo getMetaSharedTypeInfo(TypeDef typeDef, Class clz) { jitContext.registerSerializerJITCallback( () -> CompatibleSerializer.class, () -> CodecUtils.loadOrGenCompatibleCodecClass(this, cls, typeDef), - c -> - typeInfo.setSerializer( - this, newGeneratedCompatibleSerializer(cls, c, typeDef))); + c -> { + Serializer gen = newGeneratedCompatibleSerializer(cls, c, typeDef); + typeInfo.setSerializer(this, gen); + if (hasExtraFieldsSinkField(cls)) { + registerExtraFieldsSerializer(cls, typeDef.getId(), gen); + } + }); } else if (sc == null) { sc = CompatibleSerializer.class; } @@ -1305,9 +1334,17 @@ private TypeInfo getMetaSharedTypeInfo(TypeDef typeDef, Class clz) { if (StaticGeneratedStructSerializer.class.isAssignableFrom(sc)) { typeInfo.setSerializer(this, newStaticGeneratedStructSerializer(sc, cls, typeDef)); } else if (sc == CompatibleSerializer.class) { - typeInfo.setSerializer(this, new CompatibleSerializer(this, cls, typeDef)); + CompatibleSerializer cs = new CompatibleSerializer<>(this, cls, typeDef); + typeInfo.setSerializer(this, cs); + if (hasExtraFieldsSinkField(cls)) { + registerExtraFieldsSerializer(cls, typeDef.getId(), cs); + } } else if (GeneratedCompatibleSerializer.class.isAssignableFrom(sc)) { - typeInfo.setSerializer(this, newGeneratedCompatibleSerializer(cls, sc, typeDef)); + Serializer gen = newGeneratedCompatibleSerializer(cls, sc, typeDef); + typeInfo.setSerializer(this, gen); + if (hasExtraFieldsSinkField(cls)) { + registerExtraFieldsSerializer(cls, typeDef.getId(), gen); + } } else { typeInfo.setSerializer(this, Serializers.newSerializer(this, cls, sc)); } @@ -2460,6 +2497,11 @@ class ExtRegistry { final IdentityHashMap, TypeInfo> abstractTypeInfo = new IdentityHashMap<>(); final IdentityHashMap, TransformedTypeInfo[]> transformedTypeInfo = new IdentityHashMap<>(); + // (reader class, remote TypeDef id) -> reader-class serializer that replays extra fields. + // Keyed by both because one remote TypeDef can be read into several local classes, each with + // its own replay serializer + final ConcurrentIdentityMap, Map>> extraFieldsSerializers = + new ConcurrentIdentityMap<>(); // avoid potential recursive call for seq codec generation. // ex. A->field1: B, B.field1: A final Set> getClassCtx = new HashSet<>(); diff --git a/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java b/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java index 1af0432dcc..34670904da 100644 --- a/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java +++ b/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java @@ -26,6 +26,7 @@ import org.apache.fory.config.ForyBuilder; import org.apache.fory.context.ReadContext; import org.apache.fory.context.RefReader; +import org.apache.fory.context.RefWriter; import org.apache.fory.context.WriteContext; import org.apache.fory.logging.Logger; import org.apache.fory.logging.LoggerFactory; @@ -65,13 +66,16 @@ public class CompatibleSerializer extends AbstractObjectSerializer { private static final Logger LOG = LoggerFactory.getLogger(CompatibleSerializer.class); + private final TypeDef typeDef; private final SerializationFieldInfo[] allFields; private final CompatibleCollectionArrayReader.ReadAction[] allCompatibleReadActions; private final boolean hasCompatibleCollectionArrayRead; private final RecordInfo recordInfo; + private final FieldAccessor extraFieldsSinkAccessor; private Serializer serializer; private final boolean hasDefaultValues; private final DefaultValueUtils.DefaultValueField[] defaultValueFields; + private static final Object NOT_FOUND = new Object(); public CompatibleSerializer(TypeResolver typeResolver, Class type, TypeDef typeDef) { super(typeResolver, type); @@ -79,6 +83,8 @@ public CompatibleSerializer(TypeResolver typeResolver, Class type, TypeDef ty !config.checkClassVersion(), "Class version check should be disabled when compatible mode is enabled."); Preconditions.checkArgument(config.isMetaShareEnabled(), "Meta share must be enabled."); + this.typeDef = typeDef; + this.extraFieldsSinkAccessor = ForyExtraFields.findSinkAccessor(type); if (Utils.DEBUG_OUTPUT_ENABLED) { LOG.info("========== CompatibleSerializer TypeDef for {} ==========", type.getName()); LOG.info("TypeDef fieldsInfo count: {}", typeDef.getFieldCount()); @@ -216,6 +222,13 @@ private static CompatibleCollectionArrayReader.ReadAction compatibleCollectionAr @Override public void write(WriteContext writeContext, T value) { MemoryBuffer buffer = writeContext.getBuffer(); + if (extraFieldsSinkAccessor != null) { + ForyExtraFields extraField = (ForyExtraFields) extraFieldsSinkAccessor.getObject(value); + if (extraField != null && !extraField.isEmpty()) { + writeFieldsIncludingExtraFields(writeContext, value, extraField); + return; + } + } if (serializer == null) { // xlang mode will register class and create serializer in advance, it won't go to here. serializer = @@ -225,6 +238,71 @@ public void write(WriteContext writeContext, T value) { serializer.write(writeContext, value); } + /** + * Writes all fields in remote-sorted order under the remote TypeDef schema: matched fields are + * read from local getters, unmatched fields are read from the {@link ForyExtraFields} map using + * fieldInfo. + */ + private void writeFieldsIncludingExtraFields( + WriteContext writeContext, T value, ForyExtraFields extraField) { + RefWriter refWriter = writeContext.getRefWriter(); + Generics generics = writeContext.getGenerics(); + for (SerializationFieldInfo fieldInfo : allFields) { + writeFieldByCodecCategory(writeContext, value, refWriter, generics, fieldInfo, extraField); + } + } + + private void writeFieldByCodecCategory( + WriteContext writeContext, + T value, + RefWriter refWriter, + Generics generics, + SerializationFieldInfo fieldInfo, + ForyExtraFields extraField) { + + MemoryBuffer buffer = writeContext.getBuffer(); + + boolean useExtraField = false; + Object fieldValue = null; + + Object temp = extraField.getOrDefault(fieldInfo.descriptor.getName(), NOT_FOUND); + if (temp != NOT_FOUND) { + useExtraField = true; + fieldValue = temp; // might be null, which is valid + } + + switch (fieldInfo.codecCategory) { + case BUILD_IN: + if (useExtraField) { + AbstractObjectSerializer.writeBuildInFieldValue( + writeContext, typeResolver, refWriter, fieldInfo, buffer, fieldValue); + } else { + AbstractObjectSerializer.writeBuildInField( + writeContext, typeResolver, refWriter, fieldInfo, buffer, value); + } + return; + + case CONTAINER: + if (!useExtraField) { + fieldValue = fieldInfo.fieldAccessor.getObject(value); + } + writeContainerFieldValue( + writeContext, typeResolver, refWriter, generics, fieldInfo, buffer, fieldValue); + return; + + case OTHER: + if (!useExtraField) { + fieldValue = fieldInfo.fieldAccessor.getObject(value); + } + AbstractObjectSerializer.writeField( + writeContext, typeResolver, refWriter, fieldInfo, buffer, fieldValue); + return; + + default: + throw new IllegalStateException("Unknown field codec category " + fieldInfo.codecCategory); + } + } + private T newInstance() { if (!hasDefaultValues) { return newBean(); @@ -262,6 +340,16 @@ public T read(ReadContext readContext) { return targetObject; } + /** + * Puts {@code value} into the sink on {@code target}, stashing the remote TypeDef on first use. + */ + private void captureExtraField(Object target, String name, Object value) { + if (target == null) { + throw new IllegalArgumentException("Cannot capture extra field for null target"); + } + ForyExtraFields.capture(target, extraFieldsSinkAccessor, typeDef, name, value); + } + private void setFieldValue(T targetObject, SerializationFieldInfo fieldInfo, Object fieldValue) { if (fieldInfo.fieldAccessor != null) { fieldInfo.fieldAccessor.putObject(targetObject, fieldValue); @@ -285,7 +373,8 @@ private void readFields(ReadContext readContext, Object[] fields) { RefReader refReader = readContext.getRefReader(); Generics generics = readContext.getGenerics(); for (SerializationFieldInfo fieldInfo : allFields) { - fields[counter++] = readField(readContext, refReader, generics, fieldInfo, buffer, null); + fields[counter++] = + readField(readContext, refReader, generics, fieldInfo, buffer, null, false); } } @@ -323,7 +412,8 @@ private void readFieldsWithCompatibleCollectionArray(ReadContext readContext, Ob if (Utils.DEBUG_OUTPUT_ENABLED) { printFieldDebugInfo(fieldInfo, buffer); } - fields[counter++] = readField(readContext, refReader, generics, fieldInfo, buffer, action); + fields[counter++] = + readField(readContext, refReader, generics, fieldInfo, buffer, action, false); } } @@ -339,12 +429,14 @@ private void readField( if (fieldAccessor == null) { if (fieldInfo.codecCategory == FieldGroups.FieldCodecCategory.BUILD_IN) { if (fieldInfo.fieldConverter == null) { - FieldSkipper.skipField(readContext, typeResolver, refReader, fieldInfo, buffer); + readUnmatchedField( + readContext, targetObject, refReader, generics, fieldInfo, buffer, action); } else { compatibleRead(readContext, fieldInfo, targetObject); } } else { - readField(readContext, refReader, generics, fieldInfo, buffer, action); + readUnmatchedField( + readContext, targetObject, refReader, generics, fieldInfo, buffer, action); } return; } @@ -354,7 +446,8 @@ private void readField( return; } fieldAccessor.putObject( - targetObject, readField(readContext, refReader, generics, fieldInfo, buffer, action)); + targetObject, + readField(readContext, refReader, generics, fieldInfo, buffer, action, false)); } private Object readField( @@ -363,7 +456,8 @@ private Object readField( Generics generics, SerializationFieldInfo fieldInfo, MemoryBuffer buffer, - CompatibleCollectionArrayReader.ReadAction action) { + CompatibleCollectionArrayReader.ReadAction action, + boolean captureUnmatched) { if (action != null) { return CompatibleCollectionArrayReader.read(readContext, fieldInfo.refMode, action); } @@ -374,7 +468,7 @@ private Object readField( FieldConverters.readSourceScalar(readContext, fieldInfo, fieldInfo.fieldConverter); return fieldInfo.fieldConverter.convert(sourceValue); } - if (fieldInfo.fieldAccessor == null) { + if (fieldInfo.fieldAccessor == null && !captureUnmatched) { FieldSkipper.skipField(readContext, typeResolver, refReader, fieldInfo, buffer); return null; } @@ -391,6 +485,23 @@ private Object readField( } } + private void readUnmatchedField( + ReadContext readContext, + T targetObject, + RefReader refReader, + Generics generics, + SerializationFieldInfo fieldInfo, + MemoryBuffer buffer, + CompatibleCollectionArrayReader.ReadAction action) { + if (extraFieldsSinkAccessor == null) { + FieldSkipper.skipField(readContext, typeResolver, refReader, fieldInfo, buffer); + return; + } + + Object value = readField(readContext, refReader, generics, fieldInfo, buffer, action, true); + captureExtraField(targetObject, fieldInfo.descriptor.getName(), value); + } + private void printFieldDebugInfo(SerializationFieldInfo fieldInfo, MemoryBuffer buffer) { LOG.info( "[Java] read field {} of type {}, reader index {}", diff --git a/java/fory-core/src/main/java/org/apache/fory/serializer/ForyExtraFields.java b/java/fory-core/src/main/java/org/apache/fory/serializer/ForyExtraFields.java new file mode 100644 index 0000000000..6a7692dcf7 --- /dev/null +++ b/java/fory-core/src/main/java/org/apache/fory/serializer/ForyExtraFields.java @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.serializer; + +import java.lang.reflect.Field; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; +import org.apache.fory.collection.ClassValueCache; +import org.apache.fory.meta.TypeDef; +import org.apache.fory.reflect.FieldAccessor; + +/** + * Opt-in sink for compatible-mode extra fields. A class participates by declaring a field of this + * type; the serialization framework detects it, excludes it from the normal field set, and routes + * unmatched remote fields into it instead of discarding them. + * + *

The remote {@link TypeDef} is stored transiently so that on re-serialization the framework can + * emit the remote schema header and replay all fields in their original order, allowing a + * downstream peer with the full schema to recover them. + */ +public final class ForyExtraFields { + + private final Map fields = new HashMap<>(); + + public Object get(String name) { + return fields.get(name); + } + + public Object getOrDefault(String name, Object defaultValue) { + return fields.getOrDefault(name, defaultValue); + } + + public boolean isEmpty() { + return fields.isEmpty(); + } + + public boolean containsKey(String name) { + return fields.containsKey(name); + } + + Object put(String name, Object value) { + return fields.put(name, value); + } + + /** + * GC-transparent class-keyed cache: on JVM this is backed by ClassValue so entries do not prevent + * the associated Class (or its ClassLoader) from being collected. On Android/GraalVM it falls + * back to a ConcurrentHashMap. + */ + private static final ClassValueCache> SINK_CACHE = + ClassValueCache.newClassKeyCache(16); + + public static Field findSinkField(Class cls) { + return SINK_CACHE + .get(cls, () -> scanForExtraField(cls)) + .map(FieldAccessor::getField) + .orElse(null); + } + + /** Returns a {@link FieldAccessor} for the {@link ForyExtraFields} sink field on {@code cls}. */ + public static FieldAccessor findSinkAccessor(Class cls) { + return SINK_CACHE.get(cls, () -> scanForExtraField(cls)).orElse(null); + } + + private static Optional scanForExtraField(Class cls) { + for (Class c = cls; c != null && c != Object.class; c = c.getSuperclass()) { + for (Field f : c.getDeclaredFields()) { + if (ForyExtraFields.class == f.getType()) { + return Optional.of(FieldAccessor.createAccessor(f)); + } + } + } + return Optional.empty(); + } + + // Excluded from Java serialization because TypeDef is runtime metadata used + // only for Fory serialization. If a ForyExtraFields instance is serialized + // with ObjectOutputStream, this field will be null after deserialization, so + // the captured extra fields cannot later be replayed by Fory. + private transient TypeDef typeDef; + + public TypeDef getTypeDef() { + return typeDef; + } + + public static void capture( + Object target, FieldAccessor sinkAccessor, TypeDef typeDef, String name, Object value) { + ForyExtraFields extraField = (ForyExtraFields) sinkAccessor.getObject(target); + if (extraField == null) { + extraField = new ForyExtraFields(); + extraField.typeDef = typeDef; + sinkAccessor.putObject(target, extraField); + } + extraField.put(name, value); + } +} diff --git a/java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/native-image.properties b/java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/native-image.properties index f5e5f2c88e..6fc5d1a1ec 100644 --- a/java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/native-image.properties +++ b/java/fory-core/src/main/resources/META-INF/native-image/org.apache.fory/fory-core/native-image.properties @@ -350,6 +350,7 @@ Args=--features=org.apache.fory.platform.ForyGraalVMFeature \ org.apache.fory.serializer.BufferSerializers$ByteBufferSerializer,\ org.apache.fory.serializer.CompatibleSerializer,\ org.apache.fory.serializer.CompatibleLayerSerializer,\ + org.apache.fory.serializer.ForyExtraFields,\ org.apache.fory.serializer.EnumSerializer,\ org.apache.fory.serializer.ExceptionSerializers,\ org.apache.fory.serializer.ExceptionSerializers$ExceptionSerializer,\ diff --git a/java/fory-core/src/test/java/org/apache/fory/serializer/ForyExtraFieldsTest.java b/java/fory-core/src/test/java/org/apache/fory/serializer/ForyExtraFieldsTest.java new file mode 100644 index 0000000000..d53081f182 --- /dev/null +++ b/java/fory-core/src/test/java/org/apache/fory/serializer/ForyExtraFieldsTest.java @@ -0,0 +1,1089 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.serializer; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertNotNull; +import static org.testng.Assert.assertNull; +import static org.testng.Assert.assertSame; +import static org.testng.Assert.assertTrue; + +import java.util.Arrays; +import java.util.List; +import org.apache.fory.Fory; +import org.apache.fory.builder.Generated; +import org.testng.annotations.DataProvider; +import org.testng.annotations.Test; + +/** + * Tests for foryextrafields in compatible mode. + * + *

The two core scenarios are: + * + *

    + *
  1. No sink (opt-out): unmatched remote fields are silently dropped — existing behavior, + * preserved unchanged. + *
  2. Sink declared (opt-in): unmatched remote fields are captured into a {@link ForyExtraFields} + * field and can be replayed on re-serialization so a downstream peer with the full schema + * recovers them. + *
+ * + *

The feature has two independent {@code write()} implementations — the interpreter's in {@link + * CompatibleSerializer} and the generated one from {@code CompatibleCodecBuilder}. Every test runs + * against both via the {@code config} data provider, which crosses {@code codegen} (false → + * interpreter, true → generated) with {@code refTracking} (false and true) — four combinations per + * test. The two ref-sharing tests, {@link #refSharingPreservedBetweenLocalAndExtraField} and {@link + * #roundTripNestedReferenceSharing}, instead use the {@code configRefTracking} provider and run + * only with ref-tracking enabled, since identity preservation is meaningless otherwise. {@link + * #roundTripAfterJitCompilation} additionally covers the interpreter→JIT handoff under async + * compilation. + */ +public class ForyExtraFieldsTest { + + // ------------------------------------------------------------------------- + // Model classes + // ------------------------------------------------------------------------- + + public static class UpstreamPrimitive { + public int f9; + + public UpstreamPrimitive(int f9) { + this.f9 = f9; + } + } + + public static class DownstreamPrimitiveWithSink { + public ForyExtraFields extraFields; + } + + public static class Upstream { + public int f0, f1, f2, f3, f4, f5, f6, f7, f8; + public int f9; // missing on DownstreamNoSink, captured by DownstreamWithSink + + public Upstream() {} + + public Upstream(int base) { + f0 = base; + f1 = base + 1; + f2 = base + 2; + f3 = base + 3; + f4 = base + 4; + f5 = base + 5; + f6 = base + 6; + f7 = base + 7; + f8 = base + 8; + f9 = base + 9; + } + } + + /** Partial-schema peer with NO sink: f9 is silently dropped on deserialization. */ + public static class DownstreamNoSink { + public int f0, f1, f2, f3, f4, f5, f6, f7, f8; + } + + /** Partial-schema peer WITH a sink: f9 is captured into {@code extraFields}. */ + public static class DownstreamWithSink { + public int f0, f1, f2, f3, f4, f5, f6, f7, f8; + public ForyExtraFields extraFields; + } + + /** + * Peer that has an object field in addition to primitives, for ref-tracking tests. Uses {@code + * int[]} (a mutable heap object) so Fory ref-tracking preserves identity. + */ + public static class UpstreamWithRef { + public int[] shared; + public int[] alias; + public int value; + + public UpstreamWithRef() {} + + public UpstreamWithRef(int[] arr, int v) { + this.shared = arr; + this.alias = arr; // intentional alias — same array object + this.value = v; + } + } + + public static class DownstreamRefSink { + public int value; + public ForyExtraFields extraFields; + } + + public static class UpstreamMixed { + public int age; + public String name; + public int score; + + public UpstreamMixed() {} + + public UpstreamMixed(int age, String name, int score) { + this.age = age; + this.name = name; + this.score = score; + } + } + + public static class DownstreamMixedSink { + public int age; + public ForyExtraFields extraFields; + } + + public static class DownstreamTwoHopSink { + public int age; + public String name; + public ForyExtraFields extraFields; + } + + public static class UpstreamScore { + public int age; + public int score; + + public UpstreamScore() {} + + public UpstreamScore(int age, int score) { + this.age = age; + this.score = score; + } + } + + public static class UpstreamLevel { + public int age; + public int level; + + public UpstreamLevel() {} + + public UpstreamLevel(int age, int level) { + this.age = age; + this.level = level; + } + } + + public static class DownstreamAgeSink { + public int age; + public ForyExtraFields extraFields; + } + + public static class DownstreamMixedArray { + public int age; + public String name; + public int score; + + public DownstreamMixedArray() {} + + public DownstreamMixedArray(int age, String name, int score) { + this.age = age; + this.name = name; + this.score = score; + } + } + + public static class UpstreamList { + public int id; + public List values; + + public UpstreamList() {} + + public UpstreamList(int id, List values) { + this.id = id; + this.values = values; + } + } + + public static class DownstreamListWithSink { + public int id; + public ForyExtraFields extraFields; + } + + public static class Address { + public String city; + public int zip; + + public Address() {} + + public Address(String city, int zip) { + this.city = city; + this.zip = zip; + } + } + + public static class UpstreamNested { + public int age; + public Address address; + + public UpstreamNested() {} + + public UpstreamNested(int age, Address address) { + this.age = age; + this.address = address; + } + } + + public static class DownstreamNestedSink { + public int age; + + public ForyExtraFields extraFields; + } + + public static class UpstreamNestedRef { + public Address home; + public Address work; + + public UpstreamNestedRef() {} + + public UpstreamNestedRef(Address address) { + this.home = address; + this.work = address; + } + } + + public static class DownstreamNestedRefSink { + public ForyExtraFields extraFields; + } + + public static class DownstreamTotalSink { + public ForyExtraFields extraFields; + } + + public abstract static class ExtraFieldsBase { + public ForyExtraFields extraFields; + } + + public static class DownstreamInheritedSink extends ExtraFieldsBase {} + + @Test(dataProvider = "config") + public void roundTripInheritedExtraFieldSink(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + + Address address = new Address("Manhattan", 66502); + UpstreamNested original = new UpstreamNested(30, address); + + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + + DownstreamInheritedSink intermediate = + foryB.deserialize(aToBBytes, DownstreamInheritedSink.class); + + assertNotNull(intermediate.extraFields); + + Object captured = intermediate.extraFields.get("address"); + assertNotNull(captured); + assertTrue(captured instanceof Address); + + Address capturedAddress = (Address) captured; + assertEquals(capturedAddress.city, "Manhattan"); + assertEquals(capturedAddress.zip, 66502); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + + UpstreamNested recovered = foryC.deserialize(bToCBytes, UpstreamNested.class); + + assertEquals(recovered.age, original.age); + assertNotNull(recovered.address); + assertEquals(recovered.address.city, original.address.city); + assertEquals(recovered.address.zip, original.address.zip); + } + + @Test(dataProvider = "config") + public void roundTripAllFieldsCapturedInExtraField(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + + Address address = new Address("Manhattan", 66502); + UpstreamNested original = new UpstreamNested(30, address); + + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + + DownstreamTotalSink intermediate = foryB.deserialize(aToBBytes, DownstreamTotalSink.class); + + assertNotNull(intermediate.extraFields); + + Object capturedAge = intermediate.extraFields.get("age"); + assertNotNull(capturedAge); + assertEquals(capturedAge, 30); + + Object capturedAddress = intermediate.extraFields.get("address"); + assertNotNull(capturedAddress); + assertTrue(capturedAddress instanceof Address); + + Address captured = (Address) capturedAddress; + assertEquals(captured.city, "Manhattan"); + assertEquals(captured.zip, 66502); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + + UpstreamNested recovered = foryC.deserialize(bToCBytes, UpstreamNested.class); + + assertEquals(recovered.age, original.age); + assertNotNull(recovered.address); + assertEquals(recovered.address.city, original.address.city); + assertEquals(recovered.address.zip, original.address.zip); + } + + @Test(dataProvider = "config") + public void roundTripNestedExtraField(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + + Address address = new Address("Manhattan", 66502); + UpstreamNested original = new UpstreamNested(30, address); + + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + + DownstreamNestedSink intermediate = foryB.deserialize(aToBBytes, DownstreamNestedSink.class); + + assertEquals(intermediate.age, original.age); + assertNotNull(intermediate.extraFields); + + Object captured = intermediate.extraFields.get("address"); + assertNotNull(captured); + assertTrue(captured instanceof Address); + + Address capturedAddress = (Address) captured; + assertEquals(capturedAddress.city, "Manhattan"); + assertEquals(capturedAddress.zip, 66502); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + + UpstreamNested recovered = foryC.deserialize(bToCBytes, UpstreamNested.class); + + assertEquals(recovered.age, original.age); + assertNotNull(recovered.address); + assertEquals(recovered.address.city, original.address.city); + assertEquals(recovered.address.zip, original.address.zip); + } + + @Test(dataProvider = "configRefTracking") + public void roundTripNestedReferenceSharing(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + + Address address = new Address("Manhattan", 66502); + UpstreamNestedRef original = new UpstreamNestedRef(address); + + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + + DownstreamNestedRefSink intermediate = + foryB.deserialize(aToBBytes, DownstreamNestedRefSink.class); + + assertNotNull(intermediate.extraFields); + + Object capturedHome = intermediate.extraFields.get("home"); + Object capturedWork = intermediate.extraFields.get("work"); + + assertNotNull(capturedHome); + assertNotNull(capturedWork); + assertSame(capturedHome, capturedWork); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + + UpstreamNestedRef recovered = foryC.deserialize(bToCBytes, UpstreamNestedRef.class); + + assertNotNull(recovered.home); + assertNotNull(recovered.work); + assertSame(recovered.home, recovered.work); + + assertEquals(recovered.home.city, address.city); + assertEquals(recovered.home.zip, address.zip); + } + + /** + * @param codegen {@code false} → interpreter ({@link CompatibleSerializer}); {@code true} → + * generated serializer ({@code CompatibleCodecBuilder}). + * @param refTracking whether reference tracking is enabled on the built {@link Fory}. + */ + private static Fory compatibleFory(boolean codegen, boolean refTracking) { + return Fory.builder() + .withXlang(false) + .withRefTracking(refTracking) + .withCompatible(true) + .requireClassRegistration(false) + .withCodegen(codegen) + .build(); + } + + /** + * Crosses {@code codegen} (interpreter vs generated write path) with {@code refTracking} (off vs + * on) — four combinations, so every test exercises both write paths under both ref modes. + */ + @DataProvider(name = "config") + public static Object[][] config() { + return new Object[][] {{false, false}, {false, true}, {true, false}, {true, true}}; + } + + /** + * Like {@link #config} but pins {@code refTracking} on — for the ref-sharing tests, whose + * identity assertions are meaningless without reference tracking. + */ + @DataProvider(name = "configRefTracking") + public static Object[][] configRefTracking() { + return new Object[][] {{false, true}, {true, true}}; + } + + // ------------------------------------------------------------------------- + // No-sink: unmatched fields silently dropped + // ------------------------------------------------------------------------- + + @Test(dataProvider = "config") + public void deserializeIgnoresMissingFields(boolean codegen, boolean refTracking) { + Fory writer = compatibleFory(codegen, refTracking); + Upstream upstream = new Upstream(100); + byte[] bytes = writer.serialize(upstream); + + Fory reader = compatibleFory(codegen, refTracking); + DownstreamNoSink result = reader.deserialize(bytes, DownstreamNoSink.class); + + // f0..f8 preserved; f9 is silently dropped (no sink declared) + assertEquals(result.f0, upstream.f0); + assertEquals(result.f1, upstream.f1); + assertEquals(result.f2, upstream.f2); + assertEquals(result.f3, upstream.f3); + assertEquals(result.f4, upstream.f4); + assertEquals(result.f5, upstream.f5); + assertEquals(result.f6, upstream.f6); + assertEquals(result.f7, upstream.f7); + assertEquals(result.f8, upstream.f8); + } + + /** + * When the serializer handles primitive fields, it correctly converts them into their boxed + * object equivalents when storing them as objects. + */ + @Test(dataProvider = "config") + public void capturesPrimitiveAndBoxesCorrectly(boolean codegen, boolean refTracking) { + Fory writer = compatibleFory(codegen, refTracking); + UpstreamPrimitive upstream = new UpstreamPrimitive(100); + byte[] bytes = writer.serialize(upstream); + + Fory reader = compatibleFory(codegen, refTracking); + DownstreamPrimitiveWithSink result = + reader.deserialize(bytes, DownstreamPrimitiveWithSink.class); + + assertNotNull(result.extraFields); + + Object value = result.extraFields.get("f9"); + + // primitive int stored inside Object becomes Integer + assertTrue(value instanceof Integer); + assertEquals(100, value); + } + + @Test(dataProvider = "config") + public void sinkIsNullWhenNoFieldsMissing(boolean codegen, boolean refTracking) { + Fory fory = compatibleFory(codegen, refTracking); + DownstreamWithSink obj = new DownstreamWithSink(); + obj.f0 = 7; + obj.f3 = 42; + byte[] bytes = fory.serialize(obj); + DownstreamWithSink result = fory.deserialize(bytes, DownstreamWithSink.class); + assertNull(result.extraFields); + } + + // ------------------------------------------------------------------------- + // Round-trip test: A (full) → B (partial + sink) → C (full) recovers f9 + // ------------------------------------------------------------------------- + + @Test(dataProvider = "config") + public void roundTripRecoversMissingField(boolean codegen, boolean refTracking) { + // --- A: full-schema writer --- + Fory foryA = compatibleFory(codegen, refTracking); + Upstream original = new Upstream(200); + byte[] aToBBytes = foryA.serialize(original); + + // --- B: partial-schema peer with sink --- + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamWithSink intermediate = foryB.deserialize(aToBBytes, DownstreamWithSink.class); + + // B captured f9 + assertNotNull(intermediate.extraFields); + assertEquals(intermediate.extraFields.get("f9"), original.f9); + + // B re-serializes — must emit the remote (full) TypeDef + all 10 fields + byte[] bToCBytes = foryB.serialize(intermediate); + + // --- C: full-schema receiver recovers all 10 fields --- + Fory foryC = compatibleFory(codegen, refTracking); + Upstream recovered = foryC.deserialize(bToCBytes, Upstream.class); + + assertEquals(recovered.f0, original.f0); + assertEquals(recovered.f1, original.f1); + assertEquals(recovered.f2, original.f2); + assertEquals(recovered.f3, original.f3); + assertEquals(recovered.f4, original.f4); + assertEquals(recovered.f5, original.f5); + assertEquals(recovered.f6, original.f6); + assertEquals(recovered.f7, original.f7); + assertEquals(recovered.f8, original.f8); + assertEquals(recovered.f9, original.f9); + } + + // ------------------------------------------------------------------------- + // Ref-sharing: aliased objects captured in sink preserve identity + // ------------------------------------------------------------------------- + + @Test(dataProvider = "configRefTracking") + public void refSharingPreservedBetweenLocalAndExtraField(boolean codegen, boolean refTracking) { + Fory writer = compatibleFory(codegen, refTracking); + int[] arr = {1, 2, 3}; + UpstreamWithRef original = new UpstreamWithRef(arr, 99); + // original.shared == original.alias (same array instance) + byte[] bytes = writer.serialize(original); + + Fory reader = compatibleFory(codegen, refTracking); + DownstreamRefSink result = reader.deserialize(bytes, DownstreamRefSink.class); + + assertEquals(result.value, original.value); + assertNotNull(result.extraFields); + + Object capturedShared = result.extraFields.get("shared"); + Object capturedAlias = result.extraFields.get("alias"); + assertNotNull(capturedShared); + assertSame(capturedShared, capturedAlias); + } + + // ------------------------------------------------------------------------- + // multiple extra fields of mixed types + // ------------------------------------------------------------------------- + + @Test(dataProvider = "config") + public void roundTripMultipleExtraFieldsMixedTypes(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + UpstreamMixed original = new UpstreamMixed(30, "Alice", 95); + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamMixedSink intermediate = foryB.deserialize(aToBBytes, DownstreamMixedSink.class); + assertEquals(intermediate.age, original.age); + assertNotNull(intermediate.extraFields); + assertEquals(intermediate.extraFields.get("name"), original.name); + assertEquals(((Number) intermediate.extraFields.get("score")).intValue(), original.score); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + UpstreamMixed recovered = foryC.deserialize(bToCBytes, UpstreamMixed.class); + assertEquals(recovered.age, original.age); + assertEquals(recovered.name, original.name); + assertEquals(recovered.score, original.score); + } + + // ------------------------------------------------------------------------- + // null extra field value survives round-trip + // ------------------------------------------------------------------------- + + @Test(dataProvider = "config") + public void roundTripNullExtraField(boolean codegen, boolean refTracking) { + // A null String field captured in the sink must be replayed as null, not omitted. + Fory foryA = compatibleFory(codegen, refTracking); + UpstreamMixed original = new UpstreamMixed(30, null, 95); + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamMixedSink intermediate = foryB.deserialize(aToBBytes, DownstreamMixedSink.class); + assertEquals(intermediate.age, original.age); + assertNotNull(intermediate.extraFields); + assertTrue(intermediate.extraFields.containsKey("name")); + assertNull(intermediate.extraFields.get("name")); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + UpstreamMixed recovered = foryC.deserialize(bToCBytes, UpstreamMixed.class); + assertEquals(recovered.age, original.age); + assertNull(recovered.name); + assertEquals(recovered.score, original.score); + } + + // ------------------------------------------------------------------------- + // two-hop chain — each hop re-stamps the original TypeDef + // ------------------------------------------------------------------------- + + @Test(dataProvider = "config") + public void twoHopChainRecoversAllFields(boolean codegen, boolean refTracking) { + // A (full: age, name, score) → B (age only, sink) → C (age+name, sink) → D (full) + // Each hop re-serializes using the remote (A) TypeDef so the next peer can decode. + Fory foryA = compatibleFory(codegen, refTracking); + + UpstreamMixed original = new UpstreamMixed(25, "Bob", 88); + byte[] aToBBytes = foryA.serialize(original); + + // B captures name and score + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamMixedSink b = foryB.deserialize(aToBBytes, DownstreamMixedSink.class); + assertEquals(b.age, original.age); + assertNotNull(b.extraFields); + + byte[] bToCBytes = foryB.serialize(b); + + // C receives the full A-schema bytes, matches age and name, captures score + Fory foryC = compatibleFory(codegen, refTracking); + DownstreamTwoHopSink c = foryC.deserialize(bToCBytes, DownstreamTwoHopSink.class); + assertEquals(c.age, original.age); + assertEquals(c.name, original.name); + assertNotNull(c.extraFields); + assertEquals(((Number) c.extraFields.get("score")).intValue(), original.score); + assertNull(c.extraFields.get("name")); // name is matched, not captured + + byte[] cToDBytes = foryC.serialize(c); + + // D recovers all three fields + Fory foryD = compatibleFory(codegen, refTracking); + UpstreamMixed recovered = foryD.deserialize(cToDBytes, UpstreamMixed.class); + assertEquals(recovered.age, original.age); + assertEquals(recovered.name, original.name); + assertEquals(recovered.score, original.score); + } + + // ------------------------------------------------------------------------- + // Two distinct remote schemas replayed through one Fory instance + // ------------------------------------------------------------------------- + + /** + * Two {@link DownstreamAgeSink} instances whose sinks come from two *different* remote schemas + * ({@link UpstreamScore} and {@link UpstreamLevel}) both pass through the same Fory instance. + * + *

Both objects share one local class, so replay serializers are keyed by (class, remote + * TypeDef id); each object must be routed to the serializer of its own captured schema and emit + * that schema's TypeDef header, so each downstream peer recovers its own extra field. + */ + @Test(dataProvider = "config") + public void twoDistinctRemoteSchemasReplayed(boolean codegen, boolean refTracking) { + Fory foryScore = compatibleFory(codegen, refTracking); + UpstreamScore score = new UpstreamScore(30, 95); + byte[] scoreBytes = foryScore.serialize(score); + + Fory foryLevel = compatibleFory(codegen, refTracking); + UpstreamLevel level = new UpstreamLevel(25, 5); + byte[] levelBytes = foryLevel.serialize(level); + + // Same Fory instance deserializes both — both become DownstreamAgeSink but carry + // different remote TypeDefs in their sinks. + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamAgeSink fromScore = foryB.deserialize(scoreBytes, DownstreamAgeSink.class); + DownstreamAgeSink fromLevel = foryB.deserialize(levelBytes, DownstreamAgeSink.class); + + assertEquals(fromScore.age, 30); + assertEquals(fromLevel.age, 25); + assertNotNull(fromScore.extraFields); + assertNotNull(fromLevel.extraFields); + + assertNotNull(fromScore.extraFields.getTypeDef()); + assertNotNull(fromLevel.extraFields.getTypeDef()); + assertTrue( + fromScore.extraFields.getTypeDef().getId() != fromLevel.extraFields.getTypeDef().getId(), + "sinkTypeDefId must differ between the two remote schemas"); + + // each must emit its own remote TypeDef header + byte[] replayedScore = foryB.serialize(fromScore); + byte[] replayedLevel = foryB.serialize(fromLevel); + + Fory foryC = compatibleFory(codegen, refTracking); + UpstreamScore recoveredScore = foryC.deserialize(replayedScore, UpstreamScore.class); + assertEquals(recoveredScore.age, 30); + assertEquals(recoveredScore.score, 95); + + Fory foryD = compatibleFory(codegen, refTracking); + UpstreamLevel recoveredLevel = foryD.deserialize(replayedLevel, UpstreamLevel.class); + assertEquals(recoveredLevel.age, 25); + assertEquals(recoveredLevel.level, 5); + } + + // ------------------------------------------------------------------------- + // writeReplace indirection — exercises the bare WriteContext.writeNonRef(Object) overload + // ------------------------------------------------------------------------- + + /** + * A class with a {@code writeReplace} method is handled by {@link ReplaceResolveSerializer}. When + * the replacement object's runtime type differs from the wrapper's declared type, {@code + * ReplaceResolveSerializer.write} routes the replacement through {@code + * WriteContext.writeNonRef(Object)} directly + */ + public static class ReplaceWrapper implements java.io.Serializable { + public DownstreamWithSink replacement; + + public ReplaceWrapper(DownstreamWithSink replacement) { + this.replacement = replacement; + } + + private Object writeReplace() { + return replacement; + } + } + + @Test(dataProvider = "config") + public void writeNonRefReplaysExtraFields(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + Upstream original = new Upstream(300); + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamWithSink intermediate = foryB.deserialize(aToBBytes, DownstreamWithSink.class); + assertNotNull(intermediate.extraFields); + assertEquals(intermediate.extraFields.get("f9"), original.f9); + + // wrapper.writeReplace() substitutes `intermediate` (a different runtime type), forcing + // ReplaceResolveSerializer down the REPLACED_NEW_TYPE branch, which calls + // WriteContext.writeNonRef(Object) directly on `intermediate`. + ReplaceWrapper wrapper = new ReplaceWrapper(intermediate); + byte[] wrapperBytes = foryB.serialize(wrapper); + + Fory foryC = compatibleFory(codegen, refTracking); + Object recovered = foryC.deserialize(wrapperBytes); + + assertTrue(recovered instanceof Upstream); + Upstream recoveredUpstream = (Upstream) recovered; + assertEquals(recoveredUpstream.f0, original.f0); + assertEquals(recoveredUpstream.f1, original.f1); + assertEquals(recoveredUpstream.f2, original.f2); + assertEquals(recoveredUpstream.f3, original.f3); + assertEquals(recoveredUpstream.f4, original.f4); + assertEquals(recoveredUpstream.f5, original.f5); + assertEquals(recoveredUpstream.f6, original.f6); + assertEquals(recoveredUpstream.f7, original.f7); + assertEquals(recoveredUpstream.f8, original.f8); + assertEquals(recoveredUpstream.f9, original.f9); + } + + // ------------------------------------------------------------------------- + // Arrays and collections + // ------------------------------------------------------------------------- + + @Test(dataProvider = "config") + public void roundTripUnmatchedCollectionField(boolean codegen, boolean refTracking) { + Fory writer = compatibleFory(codegen, refTracking); + UpstreamList original = new UpstreamList(1, Arrays.asList(10, 20, 30)); + byte[] writerBytes = writer.serialize(original); + + Fory reader = compatibleFory(codegen, refTracking); + DownstreamListWithSink downstream = + reader.deserialize(writerBytes, DownstreamListWithSink.class); + + assertEquals(downstream.id, 1); + assertNotNull(downstream.extraFields); + + byte[] replayedBytes = reader.serialize(downstream); + + Fory verify = compatibleFory(codegen, refTracking); + UpstreamList recovered = verify.deserialize(replayedBytes, UpstreamList.class); + + assertEquals(recovered.id, original.id); + assertEquals(recovered.values, original.values); + } + + // ------------------------------------------------------------------------- + // Nested object with its own sink — exercises writeNonRef path + // ------------------------------------------------------------------------- + + public static class UpstreamOuter { + public int id; + public NestedFull nested; + + public UpstreamOuter() {} + + public UpstreamOuter(int id, NestedFull nested) { + this.id = id; + this.nested = nested; + } + } + + public static class NestedFull { + public int f0; + public int f1; + + public NestedFull() {} + + public NestedFull(int f0, int f1) { + this.f0 = f0; + this.f1 = f1; + } + } + + /** Partial-schema outer with no sink; nested has a sink. */ + public static class DownstreamOuterNoSink { + public int id; + public NestedPartialSink nested; + + public DownstreamOuterNoSink() {} + } + + /** Partial-schema nested with a sink — captures f1 from NestedFull. */ + public static class NestedPartialSink { + public int f0; + public ForyExtraFields extraFields; + + public NestedPartialSink() {} + } + + public static class DownstreamNested { + int id; + + ForyExtraFields extraFields; + } + + // ------------------------------------------------------------------------- + // async compilation — interpreter bootstraps, JIT takes over + // ------------------------------------------------------------------------- + + /** + * Exercises the interpreter→JIT handoff. Under async compilation the first serialize uses the + * interpreter {@code write()}; once compilation completes the serializer is swapped for a {@link + * Generated} one. We wait deterministically for that swap and assert the second serialize + * genuinely ran through the generated path — then confirm both wire outputs replay the captured + * field correctly. + */ + @Test(timeOut = 30000) + public void roundTripAfterJitCompilation() throws InterruptedException { + Fory foryA = compatibleFory(false, true); + UpstreamMixed original = new UpstreamMixed(40, "Carol", 77); + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = + Fory.builder() + .withXlang(false) + .withRefTracking(true) + .withCompatible(true) + .requireClassRegistration(false) + .withAsyncCompilation(true) // interpreter first, generated serializer swaps in + .build(); + + DownstreamMixedSink intermediate = foryB.deserialize(aToBBytes, DownstreamMixedSink.class); + assertNotNull(intermediate.extraFields); + + // First serialize triggers async JIT and runs through the interpreter write path. + byte[] fromInterpreter = foryB.serialize(intermediate); + + // Wait for the generated serializer to be swapped in, then serialize via the compiled path. + while (!(foryB.getTypeResolver().getSerializer(DownstreamMixedSink.class) + instanceof Generated)) { + Thread.sleep(10); + } + byte[] fromJit = foryB.serialize(intermediate); + + // Both wire outputs must recover all three fields on a full-schema peer. + for (byte[] bytes : new byte[][] {fromInterpreter, fromJit}) { + Fory foryC = compatibleFory(false, true); + UpstreamMixed recovered = foryC.deserialize(bytes, UpstreamMixed.class); + assertEquals(recovered.age, original.age); + assertEquals(recovered.name, original.name); + assertEquals(recovered.score, original.score); + } + } + + public static class OuterFull { + public int id; + public MiddleFull middle; + + public OuterFull() {} + + public OuterFull(int id, MiddleFull middle) { + this.id = id; + this.middle = middle; + } + } + + public static class MiddleFull { + public int id; + public InnerFull inner; + + public MiddleFull() {} + + public MiddleFull(int id, InnerFull inner) { + this.id = id; + this.inner = inner; + } + } + + public static class InnerFull { + public int f0; + public int f1; + + public InnerFull() {} + + public InnerFull(int f0, int f1) { + this.f0 = f0; + this.f1 = f1; + } + } + + public static class OuterPartial { + public int id; + public MiddlePartial middle; + + public OuterPartial() {} + } + + public static class MiddlePartial { + public int id; + public InnerPartialSink inner; + + public MiddlePartial() {} + } + + public static class InnerPartialSink { + public int f0; + public ForyExtraFields extraFields; + + public InnerPartialSink() {} + } + + /** + * A (full: OuterFull → MiddleFull → InnerFull) → B (partial: OuterPartial → MiddlePartial → + * InnerPartialSink) → C (full: OuterFull → MiddleFull → InnerFull). + * + *

Each nested level is partially known by B. The innermost object captures the unknown f1 + * field in its sink. On re-serialization, the local schema should be written for each object, + * while the sink preserves the remote fields. C must recover the full nested structure. passes + * through writenonref(obj,typeinfo) path + */ + @Test(dataProvider = "config") + public void roundTripTripleNestedObjectWithInnerSink(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + + InnerFull inner = new InnerFull(10, 20); + MiddleFull middle = new MiddleFull(2, inner); + OuterFull original = new OuterFull(1, middle); + + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + OuterPartial intermediate = foryB.deserialize(aToBBytes, OuterPartial.class); + + assertEquals(intermediate.id, original.id); + assertNotNull(intermediate.middle); + assertEquals(intermediate.middle.id, original.middle.id); + + assertNotNull(intermediate.middle.inner); + assertEquals(intermediate.middle.inner.f0, original.middle.inner.f0); + + assertNotNull(intermediate.middle.inner.extraFields); + assertEquals(intermediate.middle.inner.extraFields.get("f1"), original.middle.inner.f1); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + OuterFull recovered = foryC.deserialize(bToCBytes, OuterFull.class); + + // C's recovered object state: + // + // OuterFull + // id = 1 + // middle = + // MiddleFull + // id = 2 + // inner = + // InnerFull + // f0 = 10 + // f1 = 20 + // + assertEquals(recovered.id, original.id); + + assertNotNull(recovered.middle); + assertEquals(recovered.middle.id, original.middle.id); + + assertNotNull(recovered.middle.inner); + assertEquals(recovered.middle.inner.f0, original.middle.inner.f0); + assertEquals(recovered.middle.inner.f1, original.middle.inner.f1); + } + + public static class UpstreamOuterTwoNested { + public int id; + public NestedFull left; + public NestedFull right; + + public UpstreamOuterTwoNested() {} + + public UpstreamOuterTwoNested(int id, NestedFull left, NestedFull right) { + this.id = id; + this.left = left; + this.right = right; + } + } + + public static class DownstreamOuterTwoNested { + public int id; + public NestedPartialSink left; + public NestedPartialSink right; + + public DownstreamOuterTwoNested() {} + } + + /** + * A (full: OuterWithTwoNestedFulls) → B (partial: OuterWithTwoNestedSinks) → C (full: + * OuterWithTwoNestedFulls). + * + *

Two sibling nested objects have independent sinks. The serializer must preserve the sink + * belonging to each object and must not reuse the TypeDef/schema from the first nested object + * when writing the second one. + */ + // this test goes through the writenonref(obj,cached typedinfoholder) path + @Test(dataProvider = "config") + public void roundTripMultipleNestedObjectsWithOwnSinks(boolean codegen, boolean refTracking) { + Fory foryA = compatibleFory(codegen, refTracking); + + NestedFull left = new NestedFull(10, 100); + NestedFull right = new NestedFull(20, 200); + UpstreamOuterTwoNested original = new UpstreamOuterTwoNested(1, left, right); + + byte[] aToBBytes = foryA.serialize(original); + + Fory foryB = compatibleFory(codegen, refTracking); + DownstreamOuterTwoNested intermediate = + foryB.deserialize(aToBBytes, DownstreamOuterTwoNested.class); + + assertEquals(intermediate.id, original.id); + + assertNotNull(intermediate.left); + assertEquals(intermediate.left.f0, left.f0); + assertNotNull(intermediate.left.extraFields); + assertEquals(intermediate.left.extraFields.get("f1"), left.f1); + + assertNotNull(intermediate.right); + assertEquals(intermediate.right.f0, right.f0); + assertNotNull(intermediate.right.extraFields); + assertEquals(intermediate.right.extraFields.get("f1"), right.f1); + + byte[] bToCBytes = foryB.serialize(intermediate); + + Fory foryC = compatibleFory(codegen, refTracking); + UpstreamOuterTwoNested recovered = foryC.deserialize(bToCBytes, UpstreamOuterTwoNested.class); + + assertEquals(recovered.id, original.id); + + assertNotNull(recovered.left); + assertEquals(recovered.left.f0, left.f0); + assertEquals(recovered.left.f1, left.f1); + + assertNotNull(recovered.right); + assertEquals(recovered.right.f0, right.f0); + assertEquals(recovered.right.f1, right.f1); + } +}