Fix 64-bit offset/timestamp truncation on Windows in Admin/Uuid marshalling

[Piotr WOLSKI (piochelepiotr)]

What does this PR do?

Fixes a Windows-only bug where AdminClient.list_offsets() / AdminClient.delete_records() results, and Uuid.most_significant_bits() / Uuid.least_significant_bits(), silently truncate any value above 2^31 - 1 and can return a negative number.

Root cause: cfl_PyDict_SetLong() marshals a value into a Python dict via a parameter typed long. On Linux/macOS (LP64), long is 64 bits, so this is harmless there. On Windows (LLP64), long is always 32 bits, even in a 64-bit build. Every current call site passes a genuinely 64-bit int64_t value straight from librdkafka (rd_kafka_topic_partition_t.offset, ListOffsets timestamps, rd_kafka_Uuid_t halves) through this 32-bit-only helper, so on Windows the value wraps modulo 2**32 and gets reinterpreted as signed.

Fix

Adds cfl_PyDict_SetLongLong() (backed by PyLong_FromLongLong, correctly 64-bit on every platform) and switches all five cfl_PyDict_SetLong() call sites that carry a 64-bit value, plus the two direct cfl_PyLong_FromLong() calls in the Uuid getters, to the new helper. cfl_PyDict_SetLong() itself is left as-is (doc comment added) in case a genuinely 32-bit-safe caller is ever needed; nothing currently calls it for a value that needs to be 64-bit.

Motivation

A production incident: a Windows host on kafka_consumer (a Datadog Agent integration) reported broker_offset = -1533701557 where the true high-watermark offset was 2761265739 — exactly true - 2**32. TopicPartition.offset (a separate, already 64-bit-safe code path using PyLong_FromLongLong, see confluent_kafka.c:513) reported the correct value in the same run, which is what localized the defect to this one marshalling helper.

This also affects Uuid.most_significant_bits()/least_significant_bits(), AdminClient.list_offsets() timestamps, and delete_records()'s low_watermark — any consumer of these on Windows can silently receive corrupted data above ~2.1 billion.

Relates to #1696, which reports the same symptom (a negative ListOffsets timestamp on Windows) but was never triaged to a root cause.

Reproduction

The bug only manifests where sizeof(long) == 4 (Windows LLP64, or any ILP32 platform). It doesn't reproduce on the Linux x64 / macOS CI this project runs functional tests on, which is presumably why it shipped unnoticed — the Windows CI job only builds wheels, it doesn't execute the test suite (.semaphore/semaphore.yml, "Wheels: Windows" job).

To reproduce without a Windows machine, the two files below extract the exact cfl_PyDict_SetLong() implementation from confluent_kafka.c (before) and the new cfl_PyDict_SetLongLong() (after) verbatim, feed in the real incident offset (2761265739), and run them in a 32-bit-long environment (i386 Debian container — same ABI class as Windows x64 for this purpose):

// repro.c -- pre-fix, using the exact original cfl_PyDict_SetLong()
#include <Python.h>
#include <stdint.h>
#include <inttypes.h>

#define cfl_PyLong_FromLong(v) PyLong_FromLong(v)

void cfl_PyDict_SetLong(PyObject *dict, const char *name, long val) {
        PyObject *vo = cfl_PyLong_FromLong(val);
        PyDict_SetItemString(dict, name, vo);
        Py_DECREF(vo);
}

int main(void) {
        Py_Initialize();
        int64_t c_topic_partition_offset = 2761265739LL; /* AGENT-16736 */
        PyObject *kwargs = PyDict_New();
        cfl_PyDict_SetLong(kwargs, "offset", c_topic_partition_offset);
        long observed = PyLong_AsLong(PyDict_GetItemString(kwargs, "offset"));
        printf("sizeof(long)=%zu true=%" PRId64 " observed=%ld\n",
               sizeof(long), c_topic_partition_offset, observed);
        return 0;
}

# Dockerfile.i386
FROM i386/debian:bookworm
RUN apt-get update && apt-get install -y --no-install-recommends gcc python3-dev python3
WORKDIR /repro
COPY repro.c .
RUN gcc -o repro repro.c $(python3-config --cflags --ldflags --embed) -lpython3.11
CMD ["./repro"]

$ docker build --platform linux/386 -f Dockerfile.i386 -t cfk-repro .
$ docker run --rm --platform linux/386 cfk-repro
sizeof(long)=4 true=2761265739 observed=-1533701557

-1533701557 is an exact match for the incident report. Running the same offset through the patched cfl_PyDict_SetLongLong() (backed by PyLong_FromLongLong) in the identical i386 container:

sizeof(long)=4 sizeof(long long)=8 true=2761265739 observed=2761265739
FIX VERIFIED: offset survives the C->Python boundary intact.

I'm attaching a zip with both repro files (repro.c, repro_fixed.c) and their Dockerfiles so this can be re-run directly. Happy to also open a separate suggestion issue about adding an actual test-execution step to the Windows CI job, since that's the gap that let this ship — didn't want to bundle that into this fix.

Checklist

• Verified the exact truncation and the fix against the real incident offset, in an environment where sizeof(long) == 4 (see above)
• Verified no behavior change on sizeof(long) == 8 (native macOS run, unaffected)
• Updated CHANGELOG.md
• No new automated regression test — the defect is only observable when sizeof(long) == 4, which none of this project's CI targets exercise a test run on (see Reproduction section); a Python-level unit test on the existing Linux/macOS CI would pass identically before and after this fix and wouldn't guard against a regression

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ piochelepiotr
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}