[CRASH]: "main thread terminated" due to fastlock lock/unlock mismatch (KeyDB 6.3.4)

[kumarnani11]

Environment

KeyDB Version: 6.3.4, bits=64, commit=00000000, modified=0
Mode: Standalone
Platform: Linux 5.10.240-238.966.amzn2.x86_64
Containerization: Running in Kubernetes, Bitnami Helm chart
ConfigFile: /opt/bitnami/keydb/etc/keydb.conf
Deployment Context: High connection count, background saves, frequent replication

Problem Summary

KeyDB crashed with the log line main thread terminated. The crash report includes a "Guru Meditation" error, pointing to a lock/unlock mismatch in fastlock.cpp:295. The stack trace identifies a concurrency or internal locking bug, but fast memory test passed, suggesting the RAM itself is not the immediate cause. The error occurred during high load/background saving intervals.

Relevant Log Extracts

115S 29 Oct 2025 184500.980 ------------------------------------------------
115S 29 Oct 2025 184500.980 !!! Software Failure. Press left mouse button to continue
115S 29 Oct 2025 184500.980 Guru Meditation: fastlock lock/unlock mismatch for v [fastlock.cpp:295]
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 handleBlockedClientsTimeout+0x23b (0x55a505bc3dab)
keydb-server 0.0.0.0:6379 beforeSleep+aeEventLoop+0x190 (0x55a505ab5270)
keydb-server 0.0.0.0:6379 aeProcessEvents+0x3a8 (0x55a505aac068)
keydb-server 0.0.0.0:6379 aeMain+0x3e (0x55a505aac6be)
keydb-server 0.0.0.0:6379 workerThreadMain(void)+0x105 (0x55a505ac31b5)
libx86_64-linux-gnu/libc.so.6+0x891c4 (0x7f76e7f5e1c4)
libx86_64-linux-gnu/libc.so.6+clone+0x40 (0x7f76e7fddac0)
------ INFO OUTPUT ------
main thread terminated
Bio thread for job type 0 terminated
Bio thread for job type 1 terminated
Bio thread for job type 2 terminated
Fast memory test PASSED, however your memory can still be broken.
Please run a memory test for several hours if possible.
KEYDB BUG REPORT END. Make sure to include from START to END.
Please report the crash by opening an issue on github: https://github.com/JohnSully/KeyDB/issues
Suspect RAM error? Use keydb-server --test-memory to verify it

Timeline of Events (at crash)

• Multiple background saves every few seconds: RDB files ~6-11 MB.
• Notice: Detuning locks due to high load per core (values ~90–103).
• Role: active-replica, connected to master. Replication events, sync and partial resync.
• 560+ connected clients; 1 master, 1 replica.
• Server uptime: ~1.1 days; Redis buildID 37939c54a10461d4.

Replication and Persistence Details

• Partial/failure resync events logged.
• Replication backlog resizing events.
• Disk persistence and background saves completed successfully until crash.

Additional Diagnostics and Actions

• Fast memory test PASSED.
• No custom modules loaded at time of crash.
• No other fatal errors in logs.
• High lock detuning notices; possible relationship to concurrency or contention.

Steps to Reproduce

• Not reproducible on-demand but typically happens under heavy background saves + high concurrency:
• Multiple clients connected.
• Frequent background saves (RDB).
• High load per core notifications present right before crash.
• Heavy replication/persistence activity.

[keithchew]

Hi @kumarnani11

Just looking at my local copy of keydb, I have this comment:

        // Found crash when this is done, disable for now.
        // __atomic_store(&g_fHighCpuPressure, &fHighPressureNew, __ATOMIC_RELEASE);

It looks like I also encountered crashes but never had to time to find the root cause, so I disabled the detuning feature in the meantime. Maybe you could also try the same?

[kumarnani11]

Thank you for the revert. Currently, i am using the available image from Docker Hub . I need to prepare an environment to build KeyDB. It may take some time to complete the setup, and I’ll get back to you once KeyDB has been successfully built.