Improve E2E testing by self-hosting API7 in CI and using structured assertions

[guoqqqi]

Background

Current a7 E2E CI connects to an external/shared API7 environment through GitHub Secrets:

• A7_ADMIN_URL
• A7_TOKEN
• A7_GATEWAY_GROUP
• A7_GATEWAY_URL
• HTTPBIN_URL

This makes CI depend on long-lived external state and credentials. During investigation, we compared this with the adc repository. adc API7 E2E tests provision API7 inside the test flow with Docker Compose, initialize the instance, generate a test token, run sync operations, then verify resources by dumping/reading back backend state.

Proposal

Move a7 E2E tests toward a similar model:

1. Provision API7 in CI using the existing test/e2e/docker-compose.yml.
2. Initialize API7 automatically during E2E setup:
   • wait for dashboard readiness
   • log in with default admin credentials
   • upload license
   • change password if required
   • create/generate a test API token
   • resolve the default gateway group ID
3. Set test runtime values from the local environment:
   • A7_ADMIN_URL=https://localhost:7443
   • A7_TOKEN=<generated test token>
   • A7_GATEWAY_URL=http://localhost:9080
   • HTTPBIN_URL=http://localhost:8080
4. Keep an opt-in external mode, for example A7_E2E_EXTERNAL=1, for local debugging or special environments.

Comparison with adc

The useful part to borrow from adc is the testing pattern, not necessarily the exact Docker Compose topology.

For API7 backend E2E, adc only starts API7 control-plane/dashboard plus PostgreSQL. It does not deploy an API7 gateway/data-plane, dp-manager, or a local httpbin-style upstream. Its API7 tests validate control-plane state by doing sync() followed by dump() and asserting that resources were persisted correctly.

adc does have APISIX backend E2E stacks that start APISIX containers and expose data-plane/admin ports, but those are separate APISIX backend tests. They are not equivalent to a full API7 Enterprise dashboard + dp-manager + gateway + upstream traffic topology.

For a7, we should keep and improve the existing richer E2E topology in test/e2e/docker-compose.yml because the CLI has tests that benefit from real data-plane coverage:

• dashboard/control-plane APIs
• dp-manager
• API7 gateway/data-plane
• local httpbin upstream
• traffic validation through A7_GATEWAY_URL

Therefore, the intended direction is:

• borrow adc's self-hosted CI environment, automatic initialization, generated token, and write-then-read-back assertions
• do not reduce a7 to the lighter adc API7 control-plane-only compose
• preserve gateway/httpbin tests for route behavior, auth/plugin behavior, debug flows, and traffic validation

Assertion Strategy

Use a consistent E2E pattern for resource tests:

create/sync -> get/list/dump -> structured assert -> update -> get/dump assert -> delete -> get/list assert missing

For a7, the preferred validation path should be CLI-level readback:

• run a7 <resource> create/update/delete
• verify with a7 <resource> get/list -o json
• verify config operations with a7 config sync followed by a7 config dump -o json
• use direct adminAPI() checks only when the CLI does not expose the required readback or cleanup path

This is similar to adc using sync() followed by dump(), but it validates the full a7 CLI surface instead of only the backend layer.

Suggested Implementation Plan

1. Update CI to start and stop the local E2E Docker Compose stack.
2. Add bootstrap logic for API7 initialization and token generation.
3. Update existing tests that only assert stdout text to parse JSON output and assert resource fields structurally.
4. Add a core config sync -> config dump -> assert E2E test as the equivalent of the adc sync/dump flow.
5. Gradually expand resource coverage:
   • context
   • gateway-group
   • service
   • route
   • consumer
   • credential
   • ssl
   • global-rule
   • plugin-metadata
   • config sync/dump/diff
   • gateway traffic validation through local httpbin
6. After the single-version local environment is stable, add an API7 version matrix, starting small, for example current stable + dev.

Acceptance Criteria

• E2E CI no longer requires the shared demo API7 environment for the default path.
• Tests can still run against an external API7 instance when explicitly requested.
• Resource tests verify backend state with structured JSON/YAML assertions instead of relying mainly on stdout substring checks.
• config sync tests verify successful creation by reading back state with config dump or resource get/list.
• CI performs deterministic cleanup with docker compose down -v.
• Gateway/httpbin-backed tests remain available for data-plane behavior where relevant.

Open Questions

• Which API7 version should be the initial default for the local E2E stack?
• Should license upload and token generation live in Go test setup, a shell script, or a small reusable helper command?
• Should CI initially run one stable version only, or stable + dev from the start?
• Which tests should continue to support external shared environments, if any?
• Which tests should be control-plane-only, and which should require the full gateway + httpbin topology?

[guoqqqi]

Status update after PR #5 was merged (ce83883):

This issue now conflicts with the current agreed testing boundary for a7.

Current direction:

• Keep GitHub CI using the repository-configured external API7 environment (A7_ADMIN_URL and A7_TOKEN from secrets, A7_GATEWAY_GROUP=default in workflow).
• Keep test/e2e/docker-compose.yml primarily for local testing/debugging.
• Do not make full self-hosted API7 + gateway + httpbin data-plane E2E the default CI path for a7 right now.
• Do not add API7 version matrix CI at this stage.
• Focus a7 CI on CLI control-plane behavior: create/update/delete/sync via a7, then assert with get, list, or config dump.
• Leave real gateway data-plane traffic validation to the gateway repository; a7 can keep optional local/full E2E paths for debugging.

What PR #5 already completed from the broader testing direction:

• Dedicated skills validation workflow.
• validate-skills and test-skills local targets.
• Workflow split into Go Test, Go Lint, Skills Metadata & Command Validation, and e2e.
• Workflow concurrency to cancel stale CI runs.
• Updated testing strategy docs to state the current external-env CI boundary.

Recommendation: close this issue as superseded by the current boundary, or rewrite it into a narrower issue focused only on improving structured CLI readback assertions in the existing external-environment E2E flow. The self-hosted CI/bootstrap/API7 version-matrix parts should not be tracked as current work for a7.