fix(soak): repair soak tests for OTel Collector v0.151.0 bump - #1137
Draft
priyankaDhingra wants to merge 3 commits into
Draft
fix(soak): repair soak tests for OTel Collector v0.151.0 bump#1137priyankaDhingra wants to merge 3 commits into
priyankaDhingra wants to merge 3 commits into
Conversation
PR aws-observability#1135 bumped Go from 1.24.0 to 1.26.2 in go.mod files (adot/collector/lambdacomponents, opentelemetry-lambda submodule). The GitHub Actions workflows still referenced Go ~1.24.3 via actions/setup-go, causing build/runtime failures in the soak tests because the installed Go toolchain was too old for the go directive. Update all 8 workflow files to use go-version: '~1.26.2' to match the project's go.mod requirements.
…0.151.0 The OTel Collector v0.151.0 submodule bump introduced two nodejs-only breakages in the soak/integration tests: 1. Terraform "Unsupported argument": upstream nodejs/sample-apps/aws-sdk/deploy/wrapper was refactored to build layer ARNs internally from published layer versions and no longer accepts collector_layer_arn / sdk_layer_arn. Rework the integration test to instantiate terraform-aws-modules/lambda/aws + api-gateway-proxy directly and attach the locally-built layers, mirroring the python test. Go and Java deploy modules still accept the ARN inputs and are unchanged. 2. TraceValidator "data model not matched": the upstream nodejs sample app switched its AWS call from S3 ListBuckets to STS GetCallerIdentity. Update the expected trace template accordingly. Java/Go samples still use S3. sim: https://taskei.amazon.dev/tasks/PROMET-11975
…v0.151.0
The collector v0.151.0 bump surfaced a runtime INIT/invoke failure across soak jobs:
- java-agent: FUNCTION_ERROR_INIT_FAILURE during provisioned-concurrency warm-up
(surfaces as the aws_lambda_provisioned_concurrency_config waiter erroring)
- go/python/java wrappers: sample app returns HTTP 500, so no trace is produced
and TraceValidator reports "data model not matched"
Both paths share the collector extension. The ADOT config override retained
`service.telemetry.metrics.address: localhost:8888`, a field deprecated in
collector v0.123.0 and dropped by upstream in the v0.151.0 default config.
Upstream opentelemetry-collector issue #3911 documents this field causing port
conflicts and collector crashes, which would crash the Lambda extension and
produce the observed INIT/invoke failures.
Remove the telemetry block from the default ADOT collector config and the
java-agent integration test's inline config to realign with upstream v0.151.0.
NOT YET VERIFIED: soak cannot be run locally (deploys real Lambdas) and the
failing run's CloudWatch logs were already torn down. A soak run is required
to confirm this resolves the INIT failures.
sim: https://taskei.amazon.dev/tasks/PROMET-11975
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repairs the soak/integration tests that broke after the ADOT Collector v0.48.0 / OTel Collector v0.151.0 dependency bump merged in #1135 (CVE remediation, internal ref V2153606376). Stacks on top of
main(which already contains #1135).Three independent problems were found in soak run 28047386560:
~1.24.3while the module now requires 1.26.2.nodejs/sample-apps/aws-sdk/deploy/wrappermodule was refactored in v0.151.0 to build layer ARNs internally from published-layer-version inputs and no longer acceptscollector_layer_arn/sdk_layer_arn, so the integration test (which must attach locally-built layers) failed withUnsupported argument. The nodejs sample app also switched its AWS SDK call from S3ListBucketsto STSGetCallerIdentity, invalidating the expected X-Ray trace template.service.telemetry.metrics.addressfield (removed upstream for v0.151.0) is the one config element common to every failing job's collector.Changes
.github/workflows/*.yml(8)~1.24.3→~1.26.2nodejs/integration-tests/aws-sdk/wrapper/{main,outputs}.tfterraform-aws-modules/lambda/aws@7.19.0+api-gateway-proxydirectly and attach locally-built layers, mirroring the python integration testadot/utils/expected-templates/nodejs-aws-sdk-wrapper.jsonGetCallerIdentitysegment (was S3)adot/collector/config.yaml,java/integration-tests/aws-sdk/agent/main.tfservice.telemetry.metrics.address: localhost:8888Go/Java deploy modules still accept the ARN inputs and their sample apps still call S3, so they are intentionally left unchanged.
Testing
Verified locally (static):
config.yamlremains valid after removing thetelemetryblock; traces →awsxray, metrics →debug.var.*referenced in the reworked nodejsmain.tfare declared invariables.tf.lambda_function_name,lambda_function_invoke_arn,lambda_role_name,api_gateway_url) match the shared modules and the working python wrapper test.NOT yet verified (requires a live run):
terraform validate/terraform init(not run locally).FUNCTION_ERROR_INIT_FAILURE(java-agent) and runtime HTTP 500s (go/python/java-wrapper) seen in the failing soak, but this can only be confirmed by a fresh soak run. Please dispatchsoaking.ymlwith-t 10800and confirm before merge.GetCallerIdentitysegment name is inferred from the java/go template convention; confirm against a real trace.Fixes soak breakage from #1135.