Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ __pycache__/
# Runtime / logs
*.log
logs/
playgrounds/benchmarks/*/results/

# Environment / secrets
.env
Expand Down
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ playground's README states what it needs.
| [mongodb-node](playgrounds/mongodb-node/) | Node.js — MongoDB driver | Express REST API + a CRUD/compatibility suite using the native Node.js driver. |
| [beanie](playgrounds/beanie/) | Python — Beanie ODM | FastAPI REST API + a CRUD/compatibility test suite using the Beanie ODM. |
| [pymongo](playgrounds/pymongo/) | Python — PyMongo driver | Flask REST API + a CRUD/compatibility test suite using the raw PyMongo driver. |
| [benchmarks](playgrounds/benchmarks/) | Shared + adapters | Repeatable performance experiments, including local latency analysis. |

More MongoDB driver playgrounds are planned. Contributions are welcome.

Expand Down Expand Up @@ -80,6 +81,10 @@ cd playgrounds/mongoose
See the [shared telemetry guide](shared/telemetry/README.md) for image,
configuration, and verification details.

For configurable baseline and traced load experiments, use the
[local latency benchmark](playgrounds/benchmarks/local-latency/). It currently
includes a Mongoose adapter and a common contract for additional drivers.

## Repository Layout

```
Expand All @@ -89,6 +94,7 @@ documentdb-playground/
├── shared/
│ └── telemetry/ # Collector + Jaeger + tracing-enabled DocumentDB stack
└── playgrounds/
├── benchmarks/ # Performance experiments + driver adapters
├── mongoose/ # Node.js + Mongoose ODM
├── mongodb-node/ # Node.js + MongoDB native driver
├── beanie/ # Python + Beanie ODM
Expand Down
12 changes: 12 additions & 0 deletions playgrounds/benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# DocumentDB benchmarks

This directory contains repeatable synthetic performance experiments. Each
benchmark documents its workload, controls, measurement boundaries, artifacts,
and interpretation limits. Results are diagnostic and are not production
capacity claims.

## Available benchmarks

| Benchmark | Purpose |
| --- | --- |
| [Local latency](local-latency/) | Compare untraced and traced client latency, then decompose connected traces into client, gateway, and PostgreSQL segments. |
99 changes: 99 additions & 0 deletions playgrounds/benchmarks/local-latency/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# DocumentDB local latency benchmark

This playground runs the same driver-native workload twice: once without
application tracing and once with a connected application-to-gateway trace. It
stores client percentiles, throughput, trace-segment percentiles, and direct
Jaeger links in a durable experiment directory.

The initial adapter uses Mongoose. Shared orchestration invokes adapters as
processes and does not import Node.js or Python application code.

## Workload data

The `large-read` workload creates one book before measurement:

```json
{
"title": "documentdb-benchmark-target",
"author": "benchmark:<experiment-id>",
"genres": ["benchmark"],
"pages": 1
}
```

It builds a deterministic `$in` query containing that title followed by
nonmatching strings until the driver's BSON serializer reports at least
`BENCHMARK_QUERY_BYTES`. Seeding, index initialization, cleanup, and warm-up are
outside the measured interval. No production or external data is ingested.

## Concurrency

`BENCHMARK_CONCURRENCY` is the maximum number of in-flight database operations.
Each adapter uses a bounded worker pool: a worker starts its next operation only
after its previous operation finishes. Client latency uses a monotonic clock.

## Run

```bash
BENCHMARK_OPERATIONS=100 \
BENCHMARK_WARMUP=20 \
BENCHMARK_CONCURRENCY=4 \
BENCHMARK_QUERY_BYTES=4096 \
./playgrounds/benchmarks/local-latency/benchmark.sh
```

For a higher-volume experiment, increase operations, warm-up, concurrency, and
query size. Repeat longer runs before drawing conclusions from tail percentiles.

| Variable | Default | Meaning |
| --- | --- | --- |
| `BENCHMARK_ADAPTER` | `mongoose` | Adapter under `adapters/`. |
| `BENCHMARK_ID` | generated | Experiment and trace correlation ID. |
| `BENCHMARK_OPERATIONS` | `1000` | Measured operations per phase. |
| `BENCHMARK_WARMUP` | `50` | Unmeasured operations per phase. |
| `BENCHMARK_CONCURRENCY` | `10` | Maximum in-flight operations. |
| `BENCHMARK_QUERY_BYTES` | `16384` | Minimum serialized BSON query size. |
| `BENCHMARK_WORKLOAD` | `large-read` | Portable workload name. |
| `BENCHMARK_RESULTS_DIR` | `results/` | Artifact parent directory. |
| `DOCUMENTDB_IMAGE` | `ghcr.io/documentdb/documentdb/documentdb-local:trace-4fbbfcb8` | Official tracing-enabled image. |

## Artifacts and Jaeger

Every run creates an ignored `results/<experiment-id>/` directory:

| File | Contents |
| --- | --- |
| `experiment.json` | Image identity and artifact manifest. |
| `summary.md` | Human-readable client and hop summary with measurement limitations. |
| `baseline.json` | Client results with application tracing disabled. |
| `traced.json` | Client results with application tracing enabled. |
| `trace-analysis.json` | Jaeger counts, exclusions, percentiles, and links. |
| `trace-segments.csv` | One row per complete connected trace. |

The analysis includes client total, client time outside the gateway, gateway
total, PostgreSQL duration sum and interval union, and gateway residual. The
residual subtracts the interval union so overlapping database spans are not
double-counted. Open the median and p99 links and compare the client span with
`gateway.request`, `gateway.process_request`, and `postgres.execute`.

Cross-process clocks can be slightly skewed, so the analyzer does not claim to
split request and response transport accurately. `client_outside_gateway`
includes driver work, pool wait, serialization, TLS, network time, and response
decoding. Nested duration subtraction is not CPU profiling. Jaeger storage is
ephemeral; retain the JSON and CSV artifacts after stopping the stack.

The benchmark Collector profile omits the verbose debug exporter. Leave SQL
commenter disabled unless SQL-log correlation is the experiment.

## Add an adapter

Add executable `adapters/<name>.sh` plus driver-native workload code in that
driver's playground. The adapter receives the `BENCHMARK_*` variables and must
atomically write `BENCHMARK_RESULT_FILE` according to
`schemas/adapter-result.schema.json`.

For each measured traced operation, emit a client span with `benchmark.id`,
`benchmark.phase`, `benchmark.query_bytes`, `benchmark.concurrency`,
`benchmark.workload`, and `benchmark.adapter`. Inject W3C `traceparent` into the
MongoDB command `comment` so `gateway.request` becomes its child. Reusing the
shared containers alone does not provide driver-side instrumentation.
36 changes: 36 additions & 0 deletions playgrounds/benchmarks/local-latency/adapters/mongoose.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env bash

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PLAYGROUND_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
MONGOOSE_DIR="$PLAYGROUND_ROOT/mongoose"

for command_name in node npm; do
command -v "$command_name" >/dev/null || {
echo "$command_name is required by the Mongoose benchmark adapter" >&2
exit 1
}
done

for variable_name in \
BENCHMARK_ID \
BENCHMARK_PHASE \
BENCHMARK_RESULT_FILE; do
if [ -z "${!variable_name:-}" ]; then
echo "$variable_name is required by the Mongoose benchmark adapter" >&2
exit 1
fi
done

# shellcheck source=../../mongoose/scripts/lib.sh
source "$MONGOOSE_DIR/scripts/lib.sh"

if [ -z "${MONGO_URI:-}" ]; then
MONGO_URI="$(build_uri)"
export MONGO_URI
fi
export MONGO_DB="${MONGO_DB:-mongoose_benchmark}"

(cd "$MONGOOSE_DIR/app" && npm install --omit=dev --no-audit --no-fund >/dev/null)
exec node "$MONGOOSE_DIR/app/benchmark.js"
Loading
Loading