Migrate JSON ingestion specs into gem

[jwils]

Why

The JSON ingestion gem should own the JSON-schema tests that cover its extracted implementation so the gem can enforce full coverage without a SimpleCov skip.

What

• Move JSON schema generation, metadata merge, pruner, and matcher specs into elasticgraph-json_ingestion.
• Add JSON-ingestion artifact-manager integration coverage for versioned schema artifacts, version-bump enforcement, metadata refresh, and schema evolution diagnostics.
• Add focused wrapper/scalar specs for the remaining extracted JSON-ingestion paths.
• Remove the temporary SimpleCov filter for elasticgraph-json_ingestion.

Verification

• script/run_gem_specs elasticgraph-json_ingestion
• script/type_check elasticgraph-json_ingestion
• BUNDLE_GEMFILE=Gemfile bundle exec rspec --format progress spec/integration/elastic_graph/schema_definition/rake_tasks_spec.rb spec/unit/elastic_graph/schema_definition/runtime_metadata/scalar_types_by_name_spec.rb from elasticgraph-schema_definition/
• bundle exec standardrb ... on touched specs/support

script/run_gem_specs elasticgraph-schema_definition and script/run_gem_specs elasticgraph-json_ingestion both pass with 100% coverage (with the test datastore booted).

Follow-ups

• The migrated spec files keep the compact module JSONIngestion::SchemaDefinition form so that git/GitHub detects them as file moves. A follow-up PR (after this stack merges) will convert them to the standard nested module style and re-indent.
• The JSON-schema-versioning scenarios that the new schema_artifact_manager_extension_spec.rb covers have been removed from the core rake_tasks_spec.rb; the core lines they exercised are now covered by focused unit specs instead.

Stack

Current PR is marked with ->.

• #1199 Add JSON ingestion gem shell
• #1200 Add schema definition extension plumbing
• #1201 Move JSON schema helpers to json ingestion gem
• #1203 Namespace JSON schema helpers under JSON ingestion
• #1204 Add JSON ingestion indexing extensions
• #1215 Wire JSON ingestion schema extension modules
• #1205 Use JSON ingestion as schema definition extension
• #1202 Document JSON ingestion schema APIs
• -> #1224 Migrate JSON ingestion specs into gem
• #1220 Add configurable indexing event decoder

[jwils]

Addressed in 3dd2511 on this PR.

[myronmarston]

Not done reviewing but want to submit my feedback so far...

[myronmarston]

We should implement something like

elasticgraph/elasticgraph-warehouse/spec/support/warehouse_schema_support.rb

Line 20 in bde70b9

extension_modules: [SchemaDefinition::APIExtension],

so that the APIExtension is automatically added by all tests in the json-ingestion gem.

(Please defer this to a follow up PR).

[myronmarston]

Beyond the feedback below:

• Can we add elasticgraph-json_ingestion to the list of gems that run in parallel in

  elasticgraph/script/run_gem_specs

  Line 32 in ac2c10e

  if [[ "$gem" == "elasticgraph-graphql" || "$gem" == "elasticgraph-indexer" || "$gem" == "elasticgraph-schema_definition" ]]; then

  ? I measured rspec vs flatware_rspec and found that flatware makes the json-ingestion test suite quite a bit faster.
• I audited the remaining mentions of json in elasticgraph-schema_definition and here's what I found...

elasticgraph/elasticgraph-schema_definition/lib/elastic_graph/schema_definition/schema_elements/built_in_types.rb

Lines 893 to 898 in f32b634

	# Allow any JSON for this type. The list of supported types is taken from:
	#
	# https://github.com/json-schema-org/json-schema-spec/blob/draft-07/schema.json#L23-L29
	#
	# ...except we are omitting `null` here; it'll be added by the nullability decorator if the field is defined as nullable.
	# In the index we store this as a JSON string in a `keyword` field.

should be moved to

elasticgraph/elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/factory_extension.rb

Line 46 in f32b634

"Untyped" => {type: ["array", "boolean", "integer", "number", "object", "string"].freeze},

Some RBS signatures remain to migrate (or just remove if already copied over):

sig/elastic_graph/schema_definition/schema_elements/field.rbs:12:        type jsonSchema = untyped
sig/elastic_graph/schema_definition/schema_elements/type_with_subfields.rbs:58:          # ?json_schema: SchemaElements::Field::jsonSchema,
sig/elastic_graph/schema_definition/test_support.rbs:8:        ?json_schema_version: ::Integer,
sig/elastic_graph/schema_definition/test_support.rbs:20:        ?json_schema_version: ::Integer,
sig/elastic_graph/schema_definition/api.rbs:5:    type jsonSchemaLayer = :nullable | :array
sig/elastic_graph/schema_definition/api.rbs:6:    type jsonSchemaLayersArray = ::Array[jsonSchemaLayer]

(Note: these latter 2 are not really related to the PR as they aren't tests, but now that the tests have move this is the first point an audit was possible. Feel free to move those in a follow up PR).

Beyond that, this LGTM!

[myronmarston]

deleted_type, deleted_field, and renamed_from are really only needed for use with the JSON schema definition APIs. They are used to record additional metadata on the versioned JSON schema artifacts. Since they don't have any use beyond that, it probably makes sense to move them into elasticgraph-json_ingestion in a follow up PR.

[myronmarston]

This test doesn't seem related to ScalarTypeExtension at all. It seems to exist to cover some branches in factory_extension.rb. I'd recommend we move it there and simplify it drastically:

diff --git a/elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/factory_extension.rb b/elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/factory_extension.rb
index 93f2ae09..2bff3962 100644
--- a/elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/factory_extension.rb
+++ b/elasticgraph-json_ingestion/lib/elastic_graph/json_ingestion/schema_definition/factory_extension.rb
@@ -90,7 +90,7 @@ module ElasticGraph
               extended_type.json_schema(**options)
             end
 
-            yield extended_type if block_given?
+            yield extended_type
             extended_type.finalize_json_schema_configuration!
           end
         end
@@ -104,7 +104,7 @@ module ElasticGraph
         def new_type_with_subfields(schema_kind, name, wrapping_type:, field_factory:)
           super(schema_kind, name, wrapping_type: wrapping_type, field_factory: field_factory) do |type|
             extended_type = type.extend(SchemaElements::TypeWithSubfieldsExtension) # : ::ElasticGraph::SchemaDefinition::SchemaElements::TypeWithSubfields & SchemaElements::TypeWithSubfieldsExtension
-            yield extended_type if block_given?
+            yield extended_type
           end
         end
 
diff --git a/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/factory_extension_spec.rb b/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/factory_extension_spec.rb
new file mode 100644
index 00000000..7080b068
--- /dev/null
+++ b/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/factory_extension_spec.rb
@@ -0,0 +1,29 @@
+# Copyright 2024 - 2026 Block, Inc.
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+#
+# frozen_string_literal: true
+
+require "elastic_graph/constants"
+require "elastic_graph/errors"
+require "elastic_graph/json_ingestion/schema_definition/factory_extension"
+require "elastic_graph/spec_support/schema_definition_helpers"
+
+module ElasticGraph
+  module JSONIngestion
+    module SchemaDefinition
+      RSpec.describe FactoryExtension do
+        include_context "SchemaDefinitionHelpers"
+
+        it "allows `enum_type` and `interface_type` to be called without a block" do
+          define_schema(schema_element_name_form: "snake_case") do |schema|
+            schema.enum_type "EmptyEnum"
+            schema.interface_type "EmptyInterface"
+          end
+        end
+      end
+    end
+  end
+end
diff --git a/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/schema_elements/scalar_type_extension_spec.rb b/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/schema_elements/scalar_type_extension_spec.rb
index 364dfe2e..15199586 100644
--- a/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/schema_elements/scalar_type_extension_spec.rb
+++ b/elasticgraph-json_ingestion/spec/unit/elastic_graph/json_ingestion/schema_definition/schema_elements/scalar_type_extension_spec.rb
@@ -28,37 +28,6 @@ module ElasticGraph
             }.to raise_error Errors::SchemaError, a_string_including("BigInt", "lacks `json_schema`")
           end
 
-          it "extends schema elements created without customization blocks" do
-            api = build_api
-            api.enum_type "EmptyEnum"
-            api.interface_type "EmptyInterface"
-            direct_type_with_subfields = api.factory.new_type_with_subfields(
-              :object,
-              "DirectObject",
-              wrapping_type: nil,
-              field_factory: api.factory.method(:new_field)
-            )
-
-            # An enum's derived GraphQL types are built from a derived scalar twin, which can only be
-            # built if `EnumTypeExtension` configured the twin's `json_schema`; otherwise building it
-            # raises a "lacks `json_schema`" error.
-            expect {
-              api.state.enum_types_by_name.fetch("EmptyEnum").derived_graphql_types
-            }.not_to raise_error
-
-            # `json_schema` is only available on types extended with `TypeWithSubfieldsExtension`.
-            interface_type = api.state.object_types_by_name.fetch("EmptyInterface")
-            interface_type.json_schema minProperties: 1
-            expect(interface_type.json_schema_options).to eq({minProperties: 1})
-
-            direct_type_with_subfields.json_schema minProperties: 2
-            expect(direct_type_with_subfields.json_schema_options).to eq({minProperties: 2})
-
-            expect {
-              build_api.scalar_type "BigInt"
-            }.to raise_error Errors::SchemaError, a_string_including("BigInt", "lacks `json_schema`")
-          end
-
           it "infers a numeric missing-value placeholder for JSON-safe unsigned_long scalars with custom coercion" do
             grouping_missing_value_placeholder = grouping_missing_value_placeholder_for(
               "unsigned_long",
@@ -215,16 +184,6 @@ module ElasticGraph
             # when one worker runs multiple suites.
             "elastic_graph/spec_support/example_extensions/scalar_coercion_adapter"
           end
-
-          def build_api
-            schema_elements = SchemaArtifacts::RuntimeMetadata::SchemaElementNames.new(form: "snake_case")
-            ::ElasticGraph::SchemaDefinition::API.new(
-              schema_elements,
-              true,
-              extension_modules: [APIExtension],
-              output: log_device
-            )
-          end
         end
       end
     end
diff --git a/elasticgraph-schema_definition/sig/elastic_graph/schema_definition/factory.rbs b/elasticgraph-schema_definition/sig/elastic_graph/schema_definition/factory.rbs
index 0a5c325b..cc55a8a9 100644
--- a/elasticgraph-schema_definition/sig/elastic_graph/schema_definition/factory.rbs
+++ b/elasticgraph-schema_definition/sig/elastic_graph/schema_definition/factory.rbs
@@ -112,7 +112,7 @@ module ElasticGraph
         ::String,
         wrapping_type: SchemaElements::anyObjectType,
         field_factory: ::Method
-      ) ?{ (SchemaElements::TypeWithSubfields) -> void } -> SchemaElements::TypeWithSubfields
+      ) { (SchemaElements::TypeWithSubfields) -> void } -> SchemaElements::TypeWithSubfields
       @@type_with_subfields_new: ::Method
 
       def new_union_type: (::String) { (SchemaElements::UnionType) -> void } -> SchemaElements::UnionType

Note that factory.new_type_with_subfields and factory.new_scalar_type both yield unconditionally w/o the json-ingestion extension:

elasticgraph/elasticgraph-schema_definition/lib/elastic_graph/schema_definition/factory.rb

Lines 261 to 265 in ac2c10e

	def new_scalar_type(name)
	@@scalar_type_new.call(@state, name.to_s) do |scalar_type|
	yield scalar_type
	end
	end

elasticgraph/elasticgraph-schema_definition/lib/elastic_graph/schema_definition/factory.rb

Lines 283 to 287 in ac2c10e

	def new_type_with_subfields(schema_kind, name, wrapping_type:, field_factory:)
	@@type_with_subfields_new.call(schema_kind, @state, name, wrapping_type: wrapping_type, field_factory: field_factory) do |type_with_subfields|
	yield type_with_subfields
	end
	end

...so there's no need for the json-ingestion extension to support no block for these. I also noticed the type signature was wrong in factory.rbs so my suggested snippet above fixes that, too.

[myronmarston]

It could help to wrap all the ones related to grouping_missing_value_placeholder in a describe for organizational purposes.