test(query-engine): add differential PromQL testing infra (#590, #594) (#630)

* feat(promql-compliance): add remote-write data seeder (#594)

Adds promql-compliance/seeder, a standalone Go module that builds a
prompb.WriteRequest for a fixed, hand-authored dataset (counters, a
gauge, and a label-churn series designed to exercise instant-vs-range
divergence), snappy-compresses it, and pushes the same bytes via
remote write to both a reference Prometheus and ASAPQuery's own
remote-write ingest endpoint. This is the data-seeding half of the
differential PromQL compliance harness proposed in #594; the
comparison-harness half is a separate workstream.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(query-engine): add generic multi-step range/instant equivalence oracle (#590)

Adds the test #590 calls out as missing: for a stable key set, one
range(start,end,step) query's per-timestamp series must exactly equal N
separate instant(t) queries (one per step), not just checked at a single
timestamp. Covers {Tumbling, Sliding} x {single-population Sum,
dual-population Count} x >=4 steps. All four pass against current code.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(promql-compliance): add instant-query diffing to promql-compliance-tester

Vendor prometheus/compliance/promql (commit 67b8327, Apache 2.0) into
promql-compliance/harness/ and patch Comparer.Compare to also run and diff an
instant query (PromAPI.Query) alongside the existing range query, since
upstream only ever exercised QueryRange. Range and instant outcomes are
tracked and reported independently (Result.RangeSuccess/InstantSuccess), so
"PASS: range, FAIL: instant" is a representable outcome instead of one
aggregate pass/fail -- this is the exact bug class cataloged in #589.

Adds config.yaml with placeholder reference/test target URLs and four seed
regression test cases ported from the query patterns described in #589,
#583, and #584 (top-k over range, per-step key snapshot churn, and two
sliding-window rate queries). Adds comparer_test.go with fake in-process
PromAPI doubles covering both diverging directions plus error/should-fail
handling, since no unit tests existed upstream.

Part of #594.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(query-engine): add boundary/validation tests for range queries (#590)

validate_range_query_params had no dedicated tests. Adds direct unit tests
for its three error branches (start>=end, step==0, step not a multiple of
the tumbling window) plus the happy path, in an inline test module next to
the function (it's private and validate_range_query_params's error string
is discarded before reaching any public caller, so the exact-string
assertions can't be made from crate::tests).

Also adds end-to-end coverage via handle_range_query_promql confirming each
bad-param case is actually rejected in practice, including the start == end
boundary specifically, and a new test that runs the same keys-but-no-value
orphan-group scenario through both the instant and range entry points and
diffs their skip/error behavior explicitly, so a future regression that
splits their behavior fails here instead of only in one of the two existing
per-path tests.

* test(query-engine): keyed sliding-window property oracle (#590)

Adds simulate_sliding_window_keyed, a pure-function sliding-window
oracle extended to keyed/grouped data (a sibling to the existing
simulate_sliding_window/simulate_sliding_window_with_alignment, kept
separate since those two are exercised by several existing
index-slicing call sites that don't have a notion of a key). Drives it
against a small deterministic sweep of window/slide configs and
per-key presence patterns (appearing, disappearing, oscillating, gap
mid-range) through a real SimpleEngine via
create_engine_multi_timestamp_with_window +
handle_range_query_promql.

The sweep surfaced a real bug: execute_range_query_pipeline computes
each Sliding-window step's window_start as
current_time.saturating_sub(lookback_ms), so every output step before
window_size_ms worth of history exists aliases onto the store's
start=0 window instead of correctly having no sample. Captured as a
minimal, #[ignore]'d regression
(sliding_window_range_query_start_before_window_size_ms_returns_wrong_value)
rather than patched, per #590's ground rules; the main property sweep
starts each scenario at its own window_size_ms to avoid that
known-buggy region while still exercising the key-expansion/merge
behavior it's meant to check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(promql-compliance): add end-to-end quick start

* test(query-engine): add fixture-driven differential runner

* test(query-engine): stabilize differential smoke run

* test(query-engine): derive compatible differential planner timing

* docs(query-engine): document differential experiment workflow

* test(query-engine): align planner timing with query ranges

* refactor(asap-tools): remove legacy promql harness

* refactor(query-engine): move range tests to follow-up issue

* fix(asap-tools): validate differential range resolution

* fix(asap-tools): reject off-grid instant comparisons

* ci(query-engine): cache differential Docker builds

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Author: milindsrivastava1997

.github/workflows/promql-differential.yml

@@ -0,0 +1,68 @@
+name: PromQL Differential Tests
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
+    branches: [main]
+    paths:
+      - 'promql-compliance/**'
+      - 'asap-query-engine/**'
+      - 'asap-planner-rs/**'
+      - 'asap-common/**'
+      - 'Cargo.toml'
+      - 'Cargo.lock'
+      - '.github/workflows/promql-differential.yml'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+  packages: read
+
+jobs:
+  differential:
+    if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GHCR for Docker layer cache
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.x'
+
+      - name: Run seeder tests
+        working-directory: promql-compliance/seeder
+        run: go test ./...
+
+      - name: Run runner tests
+        working-directory: promql-compliance/runner
+        run: go test ./...
+
+      - name: Run live PromQL differential suite
+        working-directory: promql-compliance/runner
+        run: make run
+
+      - name: Upload differential report and service logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: promql-differential-results
+          path: |
+            promql-compliance/runner/differential-report.json
+            /tmp/asapquery-differential-*
+          if-no-files-found: warn

.gitignore

@@ -5,6 +5,8 @@ asap-quickstart/bin/
 # roborev snapshots
 /.roborev/
 
+# Differential test output
+promql-compliance/runner/differential-report.json
 /.agent/
 
 /**/*.log

promql-compliance/ARCHITECTURE.md

@@ -0,0 +1,106 @@
+# Differential experiment architecture
+
+The differential runner answers one question:
+
+> Given the same samples and the same PromQL request, does ASAPQuery produce
+> the same result as Prometheus?
+
+The experiment uses Prometheus as the reference implementation. ASAPQuery is
+the system under test.
+
+## Stack
+
+```text
+                         query requests
+                    ┌─────────────────────┐
+                    │  differential runner│
+                    └─────────┬───────────┘
+                              │
+                ┌─────────────┴─────────────┐
+                │                           │
+         Prometheus reference        ASAPQuery query API
+            host :19090                 host :18088
+                ▲                           ▲
+                │                           │
+         remote-write data          remote-write ingest
+                │                       host :19091
+                │                           ▲
+                └───────────┬───────────────┘
+                            │
+                    identical samples
+
+       ASAPQuery planner ──► shared planner-output volume ──► query engine
+```
+
+The Compose services are:
+
+| Service | Role | Host ports |
+| --- | --- | --- |
+| `prometheus` | Reference PromQL implementation and backend | `19090` |
+| `planner` | Generates ASAPQuery inference and streaming configuration | none |
+| `queryengine` | Runs ASAPQuery in precompute mode | query `18088`, ingest `19091` |
+
+Inside the Compose network, the query engine reaches Prometheus at
+`http://prometheus:9090`. The engine configuration sets
+`forward_unsupported_queries: false`, so an unsupported query is rejected
+instead of being answered by Prometheus.
+
+## Run lifecycle
+
+The runner performs these stages:
+
+1. Load a dataset fixture and a query suite.
+2. Choose a base timestamp. Dataset sample offsets and suite evaluation times
+   are relative to this timestamp.
+3. Generate temporary planner and engine configuration.
+4. Start Prometheus and wait for its health check.
+5. Run the planner. Its output is placed in a named shared volume.
+6. Start the ASAPQuery query engine after the planner completes.
+7. Send the same encoded remote-write batches to Prometheus and ASAPQuery.
+8. Wait for both query APIs to be ready and for the first probe query to return
+   samples.
+9. Execute every configured range and instant query against both targets.
+10. Write the JSON report and, by default, remove the containers, network, and
+    named volumes.
+
+Use `--keep-services` when inspecting logs or making manual requests after the
+runner exits.
+
+## Configuration boundaries
+
+The checked-in dataset and suite are inputs to the runner, not service
+configuration. The runner generates these temporary files:
+
+- `controller-config.yaml`: metrics, query groups, planner timing, and cleanup
+  policy.
+- `engine_config.yaml`: HTTP ports, backend, ingestion, logging, and paths to
+  planner output.
+
+The planner receives metric and label hints from the dataset, so it does not
+need to discover them from a separate live data source in this workflow.
+
+## Comparison model
+
+For each query, the runner can perform four related checks:
+
+- Range comparison: Prometheus range result versus ASAPQuery range result.
+- Instant comparison: Prometheus instant result versus ASAPQuery instant result.
+- Reference parity: Prometheus range-at-t versus Prometheus instant-at-t.
+- Test parity: ASAPQuery range-at-t versus ASAPQuery instant-at-t.
+
+The report passes only if every configured comparison passes. An unexpected
+HTTP/query error from either target fails the comparison, even if both targets
+fail. The exception is a query explicitly marked `expect_error: true`, where
+both targets must return an error.
+
+Equal successful empty results are still equal results; use a dataset and
+probe query that should contain samples when testing ingestion readiness.
+
+## Current limitations
+
+The runner’s readiness check currently probes the first suite query and its
+first evaluation time. It does not yet expose a general ingestion watermark or
+drain signal for proving that every asynchronous batch has finished processing.
+For the same reason, suites should currently use a supported, non-empty first
+query as their readiness probe. These are tracked as follow-up synchronization
+work.