Add configurable indexing event decoder#1220
Open
jwils wants to merge 1 commit into
Open
Conversation
f31c540 to
dbc7b07
Compare
dbc7b07 to
6b885e2
Compare
This was referenced Jun 1, 2026
c1771af to
d6dba0f
Compare
1a7aee7 to
d18af46
Compare
ff22ea0 to
ef120ae
Compare
d18af46 to
0720530
Compare
ef120ae to
f3a3547
Compare
0720530 to
bc078de
Compare
f3a3547 to
12be6cd
Compare
bc078de to
a07b547
Compare
12be6cd to
2e2996c
Compare
a07b547 to
1228bcd
Compare
2e2996c to
50e8b8c
Compare
1228bcd to
a0d1136
Compare
50e8b8c to
e243aa7
Compare
9143a79 to
62632e1
Compare
1bcda6e to
97f7ce3
Compare
62632e1 to
77689bd
Compare
bf24a0d to
76aba82
Compare
77689bd to
e3a689f
Compare
76aba82 to
cf7511b
Compare
4199bb1 to
dced6b9
Compare
cf7511b to
6e1d20f
Compare
dced6b9 to
56988ec
Compare
0a8151b to
ba4a68d
Compare
5ded965 to
e648657
Compare
ba4a68d to
a4bf0e3
Compare
e648657 to
e2feffe
Compare
2b7f5d2 to
3285c93
Compare
e2feffe to
316f4cd
Compare
3285c93 to
dcd2d33
Compare
a0d30f5 to
460d40d
Compare
dcd2d33 to
3efeb9a
Compare
3efeb9a to
69c2676
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The indexer lambda currently assumes SQS payload bodies are JSON Lines. To support ingestion sources that serialize events differently, the payload decoding step needs to be configurable before the normal ElasticGraph event validation and indexing pipeline runs.
What
indexer.indexing_event_decoderextension setting using the existing runtime extension loader pattern.ElasticGraph::Indexer::IndexingEventDecoder::JSONLinesdecoder to preserve current behavior.elasticgraph-indexer_lambdaSQS processing through the configured decoder while keeping warehouse lambda compatibility.Verification
script/type_checkbundle exec rspec elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/lambda_function_spec.rb elasticgraph-warehouse_lambda/spec/unit/elastic_graph/warehouse_lambda/lambda_function_spec.rbbundle exec standardrb elasticgraph-indexer/lib/elastic_graph/indexer.rb elasticgraph-indexer/lib/elastic_graph/indexer/config.rb elasticgraph-indexer/lib/elastic_graph/indexer/event_id.rb elasticgraph-indexer/lib/elastic_graph/indexer/indexing_event_decoder.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer/config_spec.rb elasticgraph-indexer/spec/unit/elastic_graph/indexer_spec.rb elasticgraph-indexer/spec/support/example_extensions/indexing_event_decoder.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/lambda_function.rb elasticgraph-indexer_lambda/lib/elastic_graph/indexer_lambda/sqs_processor.rb elasticgraph-indexer_lambda/spec/unit/elastic_graph/indexer_lambda/sqs_processor_spec.rbscript/update_config_artifacts --verifyStack
Current PR is marked with
->.