Add duckdb.MemoryUsage JFR periodic event for non-blocking per-instance memory sampling

JFR.md

@@ -0,0 +1,160 @@
+# JFR Memory Monitoring
+
+The driver can publish DuckDB memory-usage statistics as a periodic
+Java Flight Recorder (JFR) event, so any JFR-aware tool (JMC, `jfr`
+CLI, async-profiler, Datadog/New Relic continuous profilers, …) can
+ingest DuckDB memory metrics alongside the rest of the JVM signal.
+
+The feature is **strictly opt-in** per connection and is a silent
+no-op on JVMs without JFR support. Nothing is emitted unless the
+application both sets the JDBC property on a connection and has an
+active JFR recording that enables the event.
+
+## Enabling emission
+
+Pass `jdbc_jfr_memory_monitor=<component-id>` when opening a
+connection:
+
+```java
+Properties props = new Properties();
+props.setProperty(DuckDBDriver.JDBC_JFR_MEMORY_MONITOR, "pricing-service");
+Connection conn = DriverManager.getConnection("jdbc:duckdb:/tmp/pricing.db", props);
+```
+
+…or in the URL:
+
+```
+jdbc:duckdb:/tmp/pricing.db;jdbc_jfr_memory_monitor=pricing-service
+```
+
+The `<component-id>` is an arbitrary label chosen by the application;
+it is attached to every event as the `component` field so operators
+can attribute memory to logical components in dashboards and queries.
+
+Rules:
+
+| Property value            | Effect                                         |
+| ------------------------- | ---------------------------------------------- |
+| absent                    | no events emitted for this connection          |
+| empty string              | no events emitted (same as absent)             |
+| non-empty string          | events emitted, tagged with the given value    |
+
+The JDBC property is purely an opt-in switch and a label. It does
+**not** control whether JFR is actually recording, nor the sampling
+period — those are governed by JFR recording settings (see below).
+
+## Controlling period and enabled state
+
+Sampling rate and enabled state are JFR-native settings. Configure
+them in a `.jfc` profile, via JMC, or programmatically:
+
+```java
+try (Recording r = new Recording()) {
+    r.enable("duckdb.MemoryUsage").withPeriod(Duration.ofSeconds(1));
+    r.start();
+    // ... application work ...
+    r.stop();
+    r.dump(Path.of("app.jfr"));
+}
+```
+
+Equivalent `.jfc` snippet:
+
+```xml
+<event name="duckdb.MemoryUsage">
+  <setting name="enabled">true</setting>
+  <setting name="period">1 s</setting>
+</event>
+```
+
+When no recording enables the event, the driver performs zero work —
+the DuckDB `duckdb_memory()` query is never issued.
+
+## Event schema
+
+Event name: **`duckdb.MemoryUsage`** — one event per memory tag per
+JFR tick.
+
+| Field                   | Type   | Meaning                                                           |
+| ----------------------- | ------ | ----------------------------------------------------------------- |
+| `component`             | String | Application-supplied identifier (the JDBC property value).        |
+| `tag`                   | String | DuckDB memory tag (e.g. `BASE_TABLE`, `HASH_TABLE`, `ALLOCATOR`). |
+| `dbAddress`             | long   | Native address of the DuckDB instance — stable per-instance id.   |
+| `memoryUsageBytes`      | long   | Bytes currently allocated for this tag.                           |
+| `temporaryStorageBytes` | long   | Bytes spilled to temporary storage for this tag.                  |
+
+Plus the standard JFR fields `startTime`, `duration`, `eventThread`
+(stack traces are disabled for this event).
+
+## Attribution model
+
+The monitor is keyed on the **native DuckDB instance address**
+(exposed as `dbAddress`), not on the JDBC connection:
+
+- One sample stream per distinct DuckDB instance — no double-counting
+  of shared memory.
+- `component` is captured from the first opted-in connection to an
+  instance; later opted-in connections to the same instance do not
+  change the label.
+- The monitor is created when the first opted-in connection opens and
+  torn down when the last one closes; a subsequent `getConnection`
+  starts a fresh monitor.
+
+### When two `getConnection` calls share an instance
+
+| URL / operation                          | Same `dbAddress`?                |
+| ---------------------------------------- | -------------------------------- |
+| `jdbc:duckdb:` (unnamed in-memory)       | **No** — fresh instance per call |
+| `jdbc:duckdb::memory:<name>`             | Yes, when `<name>` matches       |
+| `jdbc:duckdb:/path/to/file.db`           | Yes, when the path matches       |
+| `conn.duplicate()`                       | Yes, always                      |
+
+Connections that share an instance share a `dbAddress` and therefore
+a single `component` label (the one supplied by the first opted-in
+connection). For per-component attribution, open each component
+against a URL that yields a fresh instance — unnamed in-memory URLs
+are the simplest choice — and give each connection a unique
+`jdbc_jfr_memory_monitor` value.
+
+## Requirements
+
+A JFR-capable JVM:
+
+- OpenJDK/HotSpot 11 and newer: JFR is included.
+- Amazon Corretto 8, OpenJDK 8u272+, and several other Java 8
+  distributions: JFR backport included (`jdk.jfr` package).
+- JVMs without `jdk.jfr` (e.g. some stripped Java 8 builds): the
+  feature is a silent no-op; the `jdbc_jfr_memory_monitor` property
+  is ignored and no classes that depend on `jdk.jfr` are loaded.
+
+No additional JVM flags are required.
+
+## Inspecting a recording
+
+With the `jfr` CLI bundled with the JDK:
+
+```
+jfr summary app.jfr | grep duckdb.MemoryUsage
+jfr print --events duckdb.MemoryUsage app.jfr | head -12
+jfr metadata app.jfr | sed -n '/class MemoryUsage/,/^}/p'
+```
+
+Or open `app.jfr` in JMC for an interactive view.
+
+## Manual verification
+
+Two shell scripts reproduce the above end-to-end and are the
+recommended way to sanity-check a new build:
+
+```
+./scripts/verify-jfr.sh          # Java >= 9
+./scripts/verify-jfr-java8.sh    # Java 8 (covers both JFR and no-JFR paths)
+```
+
+Switch the active JDK first (for example
+`sdk u java 25.0.3-amzn` or `sdk u java 8.0.462-amzn`). Each script
+builds any missing artifacts, runs the four `test_jfr_memory_event*`
+unit tests, and — on Java ≥ 9 — captures a live recording and
+verifies the event with `jfr summary` and `jfr print`. The Java 8
+script additionally asserts the JFR-less fallback by running the
+driver with `jfr.jar` stripped from the bootclasspath.

README.md

@@ -22,3 +22,5 @@ java -cp "build/release/duckdb_jdbc_tests.jar:build/release/duckdb_jdbc.jar"  or
 ```
 
 Scalar function usage examples: [UDF.MD](UDF.MD)
+
+JFR memory monitoring usage: [JFR.md](JFR.md)

duckdb_java.def

@@ -26,6 +26,7 @@ Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1connect
 Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1appender
 Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1db_1ref
 Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1destroy_1db_1ref
+Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1db_1address
 Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1extension_1type
 Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1disconnect
 Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1execute

duckdb_java.exp

@@ -23,6 +23,7 @@ _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1connect
 _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1appender
 _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1db_1ref
 _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1destroy_1db_1ref
+_Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1db_1address
 _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1extension_1type
 _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1disconnect
 _Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1execute

duckdb_java.map

@@ -25,6 +25,7 @@ DUCKDB_JAVA {
     Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1appender;
     Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1db_1ref;
     Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1destroy_1db_1ref;
+    Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1db_1address;
     Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1extension_1type;
     Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1disconnect;
     Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1execute;

scripts/verify-jfr-java8.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+#
+# Manual verification of the JFR memory-monitoring feature on Java 8.
+#
+# Usage:   ./scripts/verify-jfr-java8.sh
+# Assumes: `java`, `javac` on PATH point at JDK 8 (e.g. `sdk u java 8.0.462-amzn`).
+#          `make release` has been run so that the platform-specific native
+#          library is available; this script rebuilds only the Java jars.
+
+set -euo pipefail
+
+REPO="$(cd "$(dirname "$0")/.." && pwd)"
+RELEASE="$REPO/build/release"
+BUILD="$REPO/build/java8"
+WORK="$(mktemp -d -t duckdb-jfr-verify8.XXXXXX)"
+trap 'rm -rf "$WORK"' EXIT
+
+die()  { echo "error: $*" >&2; exit 1; }
+step() { printf '\n== %s ==\n' "$*"; }
+
+# Preconditions ---------------------------------------------------------------
+
+java_major=$(java -version 2>&1 | awk -F\" '/version/ {split($2,a,"."); print (a[1]=="1")?a[2]:a[1]; exit}')
+[[ "$java_major" == "8" ]] || die "need Java 8, got $java_major (use verify-jfr.sh for Java >= 9)"
+[[ -d "$RELEASE" ]] || die "run 'make release' first (native library not found)"
+NATIVE_LIB=$(ls "$RELEASE"/libduckdb_java.so_* 2>/dev/null | head -n1) \
+    || die "no libduckdb_java.so_* under $RELEASE"
+
+# Build Java 8 jars (idempotent) ---------------------------------------------
+
+if [[ ! -f "$BUILD/duckdb_jdbc_tests.jar" ]]; then
+    step "building Java 8 jars"
+    mkdir -p "$BUILD"
+    (cd "$BUILD" \
+        && cmake -DCMAKE_BUILD_TYPE=Release "$REPO" >/dev/null \
+        && cmake --build . --target duckdb_jdbc_tests >/dev/null)
+    cp "$BUILD/duckdb_jdbc_nolib.jar" "$BUILD/duckdb_jdbc.jar"
+    jar uf "$BUILD/duckdb_jdbc.jar" -C "$(dirname "$NATIVE_LIB")" "$(basename "$NATIVE_LIB")"
+fi
+
+JAR="$BUILD/duckdb_jdbc.jar"
+TESTS="$BUILD/duckdb_jdbc_tests.jar"
+
+# Confirm bytecode 52 (Java 8) ------------------------------------------------
+
+bc_hex=$(unzip -p "$JAR" org/duckdb/DuckDBMemoryEvent.class | od -An -N8 -tx1 | awk '{print $8}')
+[[ "$bc_hex" == "34" ]] || die "expected bytecode 0x34 (Java 8), got 0x$bc_hex"
+
+# 1. Unit tests on Java 8 + JFR ----------------------------------------------
+
+step "running JFR unit tests on Java 8 (jdk.jfr backport present)"
+java -cp "$TESTS:$JAR" org/duckdb/TestDuckDBJDBC test_jfr_memory
+
+# 2. Fallback path: jfr.jar stripped from the bootclasspath -------------------
+
+step "verifying the JFR-absent fallback path"
+cat > "$WORK/NoJfrDemo.java" <<'EOF'
+import java.lang.reflect.Method;
+import java.sql.*;
+import java.util.Properties;
+
+public class NoJfrDemo {
+    public static void main(String[] a) throws Exception {
+        try { Class.forName("jdk.jfr.FlightRecorder"); throw new AssertionError("JFR present"); }
+        catch (ClassNotFoundException ok) {}
+        Class.forName("org.duckdb.DuckDBDriver");
+        Properties p = new Properties();
+        p.setProperty("jdbc_jfr_memory_monitor", "ignored");
+        try (Connection c = DriverManager.getConnection("jdbc:duckdb:", p);
+             Statement s = c.createStatement();
+             ResultSet r = s.executeQuery("SELECT 42")) { r.next(); }
+        Method f = ClassLoader.class.getDeclaredMethod("findLoadedClass", String.class);
+        f.setAccessible(true);
+        ClassLoader cl = ClassLoader.getSystemClassLoader();
+        if (f.invoke(cl, "org.duckdb.DuckDBMemoryMonitor") != null
+         || f.invoke(cl, "org.duckdb.DuckDBMemoryEvent")   != null)
+            throw new AssertionError("JFR-dependent class was loaded");
+        System.out.println("OK");
+    }
+}
+EOF
+javac -d "$WORK" -cp "$JAR" "$WORK/NoJfrDemo.java"
+
+JRE_LIB="$JAVA_HOME/jre/lib"
+BOOT="$JRE_LIB/rt.jar:$JRE_LIB/jsse.jar:$JRE_LIB/jce.jar:$JRE_LIB/charsets.jar"
+java -Xbootclasspath:"$BOOT" -cp "$WORK:$JAR" NoJfrDemo
+
+printf '\nOK\n'

scripts/verify-jfr.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+#
+# Manual verification of the JFR memory-monitoring feature on Java >= 9.
+#
+# Usage:   ./scripts/verify-jfr.sh
+# Assumes: `java`, `javac`, `jfr` on PATH point at Java 9+ (same major version).
+#          `make release` has been run (artifacts under build/release).
+
+set -euo pipefail
+
+REPO="$(cd "$(dirname "$0")/.." && pwd)"
+JAR="$REPO/build/release/duckdb_jdbc.jar"
+TESTS="$REPO/build/release/duckdb_jdbc_tests.jar"
+WORK="$(mktemp -d -t duckdb-jfr-verify.XXXXXX)"
+trap 'rm -rf "$WORK"' EXIT
+
+die()  { echo "error: $*" >&2; exit 1; }
+step() { printf '\n== %s ==\n' "$*"; }
+
+# Preconditions ---------------------------------------------------------------
+
+java_major=$(java -version 2>&1 | awk -F\" '/version/ {split($2,a,"."); print (a[1]=="1")?a[2]:a[1]; exit}')
+[[ "$java_major" -ge 9 ]] || die "need Java >= 9, got $java_major (use verify-jfr-java8.sh for Java 8)"
+command -v jfr >/dev/null || die "'jfr' CLI not found on PATH"
+[[ -f "$JAR" && -f "$TESTS" ]] || die "build artifacts missing; run 'make release' first"
+
+echo "java $java_major -- $JAR"
+
+# 1. Unit tests ---------------------------------------------------------------
+
+step "running JFR unit tests"
+java --enable-native-access=ALL-UNNAMED \
+     -cp "$TESTS:$JAR" \
+     org/duckdb/TestDuckDBJDBC test_jfr_memory
+
+# 2. End-to-end demo + jfr CLI inspection ------------------------------------
+
+step "capturing a live recording"
+cat > "$WORK/JfrDemo.java" <<'EOF'
+import java.nio.file.*;
+import java.sql.*;
+import java.time.Duration;
+import java.util.Properties;
+import jdk.jfr.Recording;
+
+public class JfrDemo {
+    public static void main(String[] a) throws Exception {
+        try (Recording r = new Recording()) {
+            r.enable("duckdb.MemoryUsage").withPeriod(Duration.ofMillis(500));
+            r.start();
+            Properties p = new Properties();
+            p.setProperty("jdbc_jfr_memory_monitor", "verify-jfr");
+            try (Connection c = DriverManager.getConnection("jdbc:duckdb:", p);
+                 Statement s = c.createStatement()) {
+                s.execute("CREATE TABLE t AS SELECT range AS i FROM range(2000000)");
+                Thread.sleep(2000);
+            }
+            r.stop();
+            r.dump(Path.of(a[0]));
+        }
+    }
+}
+EOF
+javac -d "$WORK" -cp "$JAR" "$WORK/JfrDemo.java"
+java --enable-native-access=ALL-UNNAMED -cp "$WORK:$JAR" JfrDemo "$WORK/demo.jfr"
+
+step "jfr summary (expect a non-zero count for duckdb.MemoryUsage)"
+jfr summary "$WORK/demo.jfr" | grep duckdb.MemoryUsage \
+    || die "duckdb.MemoryUsage event not found in recording"
+
+step "first event (expect component=verify-jfr, non-zero dbAddress)"
+jfr print --events duckdb.MemoryUsage "$WORK/demo.jfr" | sed -n '1,12p'
+
+printf '\nOK\n'

src/jni/duckdb_java.cpp

@@ -95,6 +95,11 @@ jobject _duckdb_jdbc_create_db_ref(JNIEnv *env, jclass, jobject conn_ref_buf) {
 	return env->NewDirectByteBuffer(db_ref, 0);
 }
 
+jlong _duckdb_jdbc_db_address(JNIEnv *env, jclass, jobject conn_ref_buf) {
+	auto conn_ref = get_connection_ref(env, conn_ref_buf);
+	return (jlong)conn_ref->db.get();
+}
+
 void _duckdb_jdbc_destroy_db_ref(JNIEnv *env, jclass, jobject db_ref_buf) {
 	if (nullptr == db_ref_buf) {
 		return;

src/jni/functions.cpp

@@ -37,6 +37,16 @@ JNIEXPORT jobject JNICALL Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1db_
 	}
 }
 
+JNIEXPORT jlong JNICALL Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1db_1address(JNIEnv * env, jclass param0, jobject param1) {
+	try {
+		return _duckdb_jdbc_db_address(env, param0, param1);
+	} catch (const std::exception &e) {
+		duckdb::ErrorData error(e);
+		ThrowJNI(env, error.Message().c_str());
+
+	}
+}
+
 JNIEXPORT void JNICALL Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1destroy_1db_1ref(JNIEnv * env, jclass param0, jobject param1) {
 	try {
 		return _duckdb_jdbc_destroy_db_ref(env, param0, param1);

src/jni/functions.hpp

@@ -21,6 +21,10 @@ jobject _duckdb_jdbc_create_db_ref(JNIEnv * env, jclass param0, jobject param1);
 
 JNIEXPORT jobject JNICALL Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1create_1db_1ref(JNIEnv * env, jclass param0, jobject param1);
 
+jlong _duckdb_jdbc_db_address(JNIEnv * env, jclass param0, jobject param1);
+
+JNIEXPORT jlong JNICALL Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1db_1address(JNIEnv * env, jclass param0, jobject param1);
+
 void _duckdb_jdbc_destroy_db_ref(JNIEnv * env, jclass param0, jobject param1);
 
 JNIEXPORT void JNICALL Java_org_duckdb_DuckDBNative_duckdb_1jdbc_1destroy_1db_1ref(JNIEnv * env, jclass param0, jobject param1);