Test relocating kotlin-reflect library

[Goooler]

Refs #1622

[Goooler]

build.gradle.kts:

plugins {
    kotlin("jvm") version "2.2.0"
    id("com.gradleup.shadow") version "9.1.0"
}

dependencies {
    implementation("org.jetbrains.kotlin:kotlin-reflect:2.2.0")
}

tasks.shadowJar {
    manifest {
        attributes["Main-Class"] = "org.example.MainKt"
    }
    enableAutoRelocation = true
}

src/main/kotlin/Main.kt:

package org.example

import kotlin.reflect.full.memberProperties

fun main() {
    println(MemberClass()::class.memberProperties)
}

private class MemberClass {
    val prop1 = "property 1"
    val prop2 by lazy { "property 2" }
}

[Goooler]

Error occurred when running command line: Exception in thread "main" java.lang.AssertionError: Built-in class shadow.kotlin.Any is not found
  at shadow.kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns$3.invoke(KotlinBuiltIns.java:93)
  at shadow.kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns$3.invoke(KotlinBuiltIns.java:88)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$MapBasedMemoizedFunction.invoke(LockBasedStorageManager.java:578)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$MapBasedMemoizedFunctionToNotNull.invoke(LockBasedStorageManager.java:681)
  at shadow.kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns.getBuiltInClassByName(KotlinBuiltIns.java:223)
  at shadow.kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns.getAny(KotlinBuiltIns.java:228)
  at shadow.kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns.getAnyType(KotlinBuiltIns.java:504)
  at shadow.kotlin.reflect.jvm.internal.impl.descriptors.NotFoundClasses$MockClassDescriptor.<init>(NotFoundClasses.kt:62)
  at shadow.kotlin.reflect.jvm.internal.impl.descriptors.NotFoundClasses.classes$lambda$2(NotFoundClasses.kt:45)
  at shadow.kotlin.reflect.jvm.internal.impl.descriptors.NotFoundClasses.accessor$NotFoundClasses$lambda1(NotFoundClasses.kt)
  at shadow.kotlin.reflect.jvm.internal.impl.descriptors.NotFoundClasses$$Lambda$1.invoke(Unknown Source)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$MapBasedMemoizedFunction.invoke(LockBasedStorageManager.java:578)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$MapBasedMemoizedFunctionToNotNull.invoke(LockBasedStorageManager.java:681)
  at shadow.kotlin.reflect.jvm.internal.impl.descriptors.NotFoundClasses.getClass(NotFoundClasses.kt:97)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.TypeDeserializer.typeConstructor$notFoundClass(TypeDeserializer.kt:151)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.TypeDeserializer.typeConstructor(TypeDeserializer.kt:156)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.TypeDeserializer.simpleType(TypeDeserializer.kt:91)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.TypeDeserializer.type(TypeDeserializer.kt:68)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassTypeConstructor.computeSupertypes(DeserializedClassDescriptor.kt:242)
  at shadow.kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor.supertypes$lambda$0(AbstractTypeConstructor.kt:78)
  at shadow.kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor.accessor$AbstractTypeConstructor$lambda0(AbstractTypeConstructor.kt)
  at shadow.kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor$$Lambda$0.invoke(Unknown Source)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValueWithPostCompute.invoke(LockBasedStorageManager.java:481)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedNotNullLazyValueWithPostCompute.invoke(LockBasedStorageManager.java:512)
  at shadow.kotlin.reflect.jvm.internal.impl.types.AbstractTypeConstructor.getSupertypes(AbstractTypeConstructor.kt:27)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope.getNonDeclaredVariableNames(DeserializedClassDescriptor.kt:360)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.variableNames_delegate$lambda$9(DeserializedMemberScope.kt:262)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.accessor$DeserializedMemberScope$OptimizedImplementation$lambda4(DeserializedMemberScope.kt)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation$$Lambda$4.invoke(Unknown Source)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedNotNullLazyValue.invoke(LockBasedStorageManager.java:527)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.StorageKt.getValue(storage.kt:42)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.getVariableNames(DeserializedMemberScope.kt:261)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope$OptimizedImplementation.addFunctionsAndPropertiesTo(DeserializedMemberScope.kt:349)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedMemberScope.computeDescriptors(DeserializedMemberScope.kt:115)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope.allDescriptors$lambda$2(DeserializedClassDescriptor.kt:279)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope.accessor$DeserializedClassDescriptor$DeserializedClassMemberScope$lambda1(DeserializedClassDescriptor.kt)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope$$Lambda$1.invoke(Unknown Source)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedLazyValue.invoke(LockBasedStorageManager.java:408)
  at shadow.kotlin.reflect.jvm.internal.impl.storage.LockBasedStorageManager$LockBasedNotNullLazyValue.invoke(LockBasedStorageManager.java:527)
  at shadow.kotlin.reflect.jvm.internal.impl.serialization.deserialization.descriptors.DeserializedClassDescriptor$DeserializedClassMemberScope.getContributedDescriptors(DeserializedClassDescriptor.kt:289)
  at shadow.kotlin.reflect.jvm.internal.impl.resolve.scopes.ResolutionScope$DefaultImpls.getContributedDescriptors$default(ResolutionScope.kt:50)
  at shadow.kotlin.reflect.jvm.internal.KDeclarationContainerImpl.getMembers(KDeclarationContainerImpl.kt:58)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data.declaredNonStaticMembers_delegate$lambda$24(KClassImpl.kt:206)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data.accessor$KClassImpl$Data$lambda11(KClassImpl.kt)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data$$Lambda$11.invoke(Unknown Source)
  at shadow.kotlin.reflect.jvm.internal.ReflectProperties$LazySoftVal.invoke(ReflectProperties.java:70)
  at shadow.kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:32)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data.getDeclaredNonStaticMembers(KClassImpl.kt:206)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data.allNonStaticMembers_delegate$lambda$28(KClassImpl.kt:215)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data.accessor$KClassImpl$Data$lambda15(KClassImpl.kt)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data$$Lambda$15.invoke(Unknown Source)
  at shadow.kotlin.reflect.jvm.internal.ReflectProperties$LazySoftVal.invoke(ReflectProperties.java:70)
  at shadow.kotlin.reflect.jvm.internal.ReflectProperties$Val.getValue(ReflectProperties.java:32)
  at shadow.kotlin.reflect.jvm.internal.KClassImpl$Data.getAllNonStaticMembers(KClassImpl.kt:215)
  at shadow.kotlin.reflect.full.KClasses.getMemberProperties(KClasses.kt:146)
  at my.MainKt.main(Main.kt:5)
  at my.MainKt.main(Main.kt)

Not all reflection functions are failed; LoadingState::class.sealedSubclasses works.

[Goooler]

@shanshin

This is a new issue based on #1539. Relocating Kotlin stdlibs, including @Metadata and module files, looks good for now. But if the kotlin-reflect library is relocated as well, there is a new error, like the stacktrace listed. It seems due to something wrong with kotlin.kotlin_builtins, any idea to fix? I didn't find a public way to modify or relocate properties in the builtin files.

[shanshin]

But if the kotlin-reflect library is relocated as well, there is a new error, like the stacktrace listed. It seems due to something wrong with kotlin.kotlin_builtins, any idea to fix? I didn't find a public way to modify or relocate properties in the builtin files.

I do not know the specifics of kotlin-reflect, however, judging by the error, the BUILT_INS_PACKAGE_NAME constant was changed during the relocation (also checked it locally).
Built-ins are classes whose name is fixed in the compiler, and it seems logical that such classes should not be relocated.

[Goooler]

Yeah, it's relocated to something like

public static final Name BUILT_INS_PACKAGE_NAME = Name.identifier("shadow.kotlin");

We can't update kotlin.kotlin_builtins in public ways, and I didn't find an easy way to prevent relocating BUILT_INS_PACKAGE_NAME in RelocatorRemapper.

Should we suggest users not relocate kotlin.reflect packages instead of patching them from Shadow side?

[Goooler]

OLD: old.jar
NEW: new.jar

 JAR   │ old      │ new      │ diff
───────┼──────────┼──────────┼──────
 class │ 12.6 MiB │ 12.6 MiB │  0 B
 other │ 74.6 KiB │ 74.6 KiB │  0 B
───────┼──────────┼──────────┼──────
 total │ 12.7 MiB │ 12.7 MiB │  0 B

 CLASSES │ old   │ new   │ diff
─────────┼───────┼───────┼───────────
 classes │  3149 │  3149 │ 0 (+0 -0)
 methods │ 27835 │ 27835 │ 0 (+0 -0)
  fields │  6257 │  6257 │ 0 (+0 -0)

=================
====   JAR   ====
=================

 size     │ diff      │ path
──────────┼───────────┼────────────────────────────────────────
          │ -28.7 KiB │ - shadow/kotlin/kotlin.kotlin_builtins
 28.7 KiB │ +28.7 KiB │ + kotlin/kotlin.kotlin_builtins
      0 B │       0 B │ + kotlin/
──────────┼───────────┼────────────────────────────────────────
 28.7 KiB │       0 B │ (total)

Name nameIdentifier18 = Name.identifier("kotlin");
Intrinsics.checkNotNullExpressionValue(nameIdentifier18, "identifier(...)");
BUILT_INS_PACKAGE_NAME = nameIdentifier18;

[Goooler]

Or we should suggest users exclude all builtin files by **/kotlin.kotlin_builtins instead such one.

[shanshin]

@Goooler, are there any details for which cases it becomes necessary to shadow the kotlin-reflect?
Is it an application or a library, JVM or Android, is it necessary to transfer stdlib along with kotlin-reflect?

When configured as in the functional test, it turns out that the kotlin/kotlin.kotlin_builtins file will be present in both stdlib and shadowed kotlin-reflect. And if their versions differ, this can have unpredictable consequences.

[Goooler]

#1622

The Jetbrains Exposed was relocated by auto relocation (adding shadow prefixes for all packages), and it's transitive dependencies were relocated too...

[shanshin]

The Jetbrains Exposed was relocated by auto relocation (adding shadow prefixes for all packages), and it's transitive dependencies were relocated too...

In general case, we should not shadow stdlib, because for many standard Kotlin classes, their name is fixed and protected from relocation (like here).
The stability of simultaneously relocated stdlib and kotlin-reflect is not guaranteed, although it is possible that this will work for some cases.

At the same time, kotlin-reflect has requirements for stdlib version.
The stdlib version in the final application must be greater or equal to the shadowed kotlin-reflect version. Stable operation on older versions of stdlib is not guaranteed.

[shanshin]

@Goooler, Summing up, I can say: Neither shadow plugin nor Kotlin are ready for relocation of kotlin-stdlib and kotlin-reflect at the same time.

• Shadow plugin can't relocate names in .kotlin_builtins file, which is necessary for kotlin-reflect operation
• If we do not relocate kotlin.* classes, then such relocation looks of little use, because in this case there is a chance of a collision with classes from the kotlin-stdlib in final app
• If shadow plugin learns to relocate names in .kotlin_builtins, even then kotlin-reflect can't work stable because names of standard classes are hardcoded in it

[Goooler]

Thanks for the help!

I document this point in #1760. Would you mind taking a review?

[Goooler]

One more thing, should we send a PR to update https://github.com/JetBrains/kotlin/blob/6511b4c13b322b5ecebcb0932d282539ba37bb39/core/compiler.common/src/org/jetbrains/kotlin/builtins/StandardNames.kt#L72 to align the style of https://github.com/JetBrains/kotlin/blob/dcd24449718cba21bd86428e5cddb9b25e5612af/core/metadata.jvm/src/org/jetbrains/kotlin/metadata/jvm/deserialization/ClassMapperLite.kt#L17.

val BUILT_INS_PACKAGE_NAME = Name.identifier(listOf('k', 'o', 't', 'l', 'i', 'n').joinToString(separator = ""))

This should help avoid some relocation issues.