[FLINK-39923][state] Fix RocksDB Statistics native memory leak

[leekeiabstraction]

What changes were proposed in this pull request?

Fix RocksDBNativeMetricMonitor.close() to invoke Statistics.close() instead of just nulling. Also hoist dbOptions.statistics() into a local in RocksDBHandle.loadDb() so a monitor-constructor failure doesn't orphan the wrapper.

Why are the changes needed?

Fix memory leak. When any of the 11 RocksDB ticker metrics is enabled, the TaskManager leaks native memory in proportion to the number of keyed state backend rebuilds e.g. during continuously failing and restarting job.

1. Latent Flink-side bug was introduced in FLINK-24786 when Statistics object was added without explicit close() on the object. This was latent as it relied on finalize() running to call dispose() and close().
2. rocksdb side finalizer was removed in facebook/rocksdb@99d8625
3. Flink 2.0+ uses frocksdb 8.10.0. Leak started occurring as close is no longer called.

Verifying this change

Reproduced on flink:2.2.2, TaskManager container limited to 4 GB cgroup matching taskmanager.memory.process.size. Jobs that throw NumberFormatException on every row is submitted with restart-strategy.fixed-delay.delay=100ms and all 11 ticker metrics enabled.

Run	Stats	Outcome
Unpatched	ON	TM OOMKilled at 4 GB in ~80 s
Patched	ON	Ran for 4 minutes, no OOM, anon RSS stayed at around 2GB

See here for reproduction steps: https://github.com/leekeiabstraction/flink/tree/reproduce-rocksdb-statistics-leak/reproduce-rocksdb-statistics-leak

Does this PR introduce any user-facing change?

None.

[flinkbot]

CI report:

• ee327e1 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[snuyanzin]

closeQuietly has it's own internal non null check
given that, can it be simplified to just

        IOUtils.closeQuietly(nativeMetricMonitor);
        IOUtils.closeQuietly(statistics);

?

[rkhachatryan]

IIUC that might cause double close of statistics (here and in nativeMetricMonitor).

[snuyanzin]

will it be an issue (double closing), asking since closeQuietly should catch all the exceptions?
Or we should keep unclosed in some cases?

anyway it is just a nit pick, so feel free to merge

[rkhachatryan]

LGTM, thanks for the fix.

FYI: @Zakelly, @fredia, @zoltar9264