Refactor/improve parallel writing throughput#92
Conversation
… compression and limiting the amount of lock time for parallel writers
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a more granular ingestion mechanism for parallel writing in the CBQ format. By splitting the ingestion process into ingest_completed (which drains already-compressed blocks off-lock) and ingest_incompleted (which flushes the residual incomplete block at thread completion), worker threads can continue accumulating records across batches without premature compression. Corresponding lifecycle hooks (on_thread_complete) have been added to the parallel reader and processor traits, along with comprehensive unit tests. The review feedback suggests two excellent performance optimizations: adding early-return checks in both ingest_completed (when there are no headers) and ingest_incompleted (when there are no records) to avoid redundant operations.
1 participant
No description provided.