Skip to content

Add configurable indexing event decoder#1220

Open
jwils wants to merge 1 commit into
joshuaw/json-ingestion-test-migrationfrom
joshuaw/indexer-ingestion-adapters
Open

Add configurable indexing event decoder#1220
jwils wants to merge 1 commit into
joshuaw/json-ingestion-test-migrationfrom
joshuaw/indexer-ingestion-adapters

Conversation

@jwils

@jwils jwils commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Why

The indexer lambda currently assumes SQS payload bodies are JSON Lines. To support ingestion sources that serialize events differently, the payload decoding step needs to be configurable before the normal ElasticGraph event validation and indexing pipeline runs.

What

  • Add an indexer.indexing_event_decoder extension setting using the existing runtime extension loader pattern.
  • Add a default ElasticGraph::Indexer::IndexingEventDecoder::JSONLines decoder to preserve current behavior.
  • Wire elasticgraph-indexer_lambda SQS processing through the configured decoder while keeping warehouse lambda compatibility.
  • Add RBS, focused specs, README docs, and regenerated config schema.

Verification

  • script/type_check
  • bundle exec rspec elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/lambda_function_spec.rb elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/lambda_function_spec.rb
  • bundle exec standardrb elasticgraph-indexer/lib/elastic_graph/indexer.rb elasticgraph-indexer/lib/elastic_graph/indexer/config.rb elasticgraph-indexer/lib/elastic_graph/indexer/event_id.rb elasticgraph-indexer/lib/elastic_graph/indexer/indexing_event_decoder.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer/spec/support/example_extensions/indexing_event_decoder.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/lambda_function.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/sqs_processor.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb
  • script/update_config_artifacts --verify

Stack

Current PR is marked with ->.

@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from f31c540 to dbc7b07 Compare May 31, 2026 05:24
@CLAassistant

CLAassistant commented May 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from dbc7b07 to 6b885e2 Compare May 31, 2026 23:52
@jwils jwils changed the base branch from main to joshuaw/json-ingestion-test-migration May 31, 2026 23:53
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from c1771af to d6dba0f Compare June 1, 2026 18:28
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from 1a7aee7 to d18af46 Compare June 1, 2026 18:42
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch 2 times, most recently from ff22ea0 to ef120ae Compare June 1, 2026 18:58
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from d18af46 to 0720530 Compare June 1, 2026 18:59
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from ef120ae to f3a3547 Compare June 1, 2026 19:01
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 0720530 to bc078de Compare June 1, 2026 19:01
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from f3a3547 to 12be6cd Compare June 4, 2026 13:59
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from bc078de to a07b547 Compare June 4, 2026 14:00
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 12be6cd to 2e2996c Compare June 4, 2026 14:19
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from a07b547 to 1228bcd Compare June 4, 2026 14:19
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 2e2996c to 50e8b8c Compare June 5, 2026 18:36
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 1228bcd to a0d1136 Compare June 5, 2026 18:36
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 50e8b8c to e243aa7 Compare June 5, 2026 18:45
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 9143a79 to 62632e1 Compare June 5, 2026 19:12
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 1bcda6e to 97f7ce3 Compare June 5, 2026 19:12
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 62632e1 to 77689bd Compare June 5, 2026 19:28
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from bf24a0d to 76aba82 Compare June 5, 2026 19:43
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from 77689bd to e3a689f Compare June 5, 2026 19:43
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 76aba82 to cf7511b Compare June 5, 2026 20:55
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch 2 times, most recently from 4199bb1 to dced6b9 Compare June 5, 2026 21:11
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from cf7511b to 6e1d20f Compare June 5, 2026 21:11
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from dced6b9 to 56988ec Compare June 5, 2026 21:29
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from 0a8151b to ba4a68d Compare June 5, 2026 21:49
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch 2 times, most recently from 5ded965 to e648657 Compare June 5, 2026 21:56
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from ba4a68d to a4bf0e3 Compare June 5, 2026 21:56
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from e648657 to e2feffe Compare June 5, 2026 22:08
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch 2 times, most recently from 2b7f5d2 to 3285c93 Compare June 5, 2026 22:35
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from e2feffe to 316f4cd Compare June 5, 2026 22:35
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 3285c93 to dcd2d33 Compare June 5, 2026 22:43
@jwils jwils force-pushed the joshuaw/json-ingestion-test-migration branch from a0d30f5 to 460d40d Compare June 5, 2026 23:37
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from dcd2d33 to 3efeb9a Compare June 5, 2026 23:37
@jwils jwils force-pushed the joshuaw/indexer-ingestion-adapters branch from 3efeb9a to 69c2676 Compare June 6, 2026 00:25
@jwils jwils marked this pull request as ready for review June 8, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants