out_kinesis_streams: add compression_level option

[ivan-bushmarinov]

Hi fluent-bit maintainers, here at Perplexity we emit records of significant size (100s of K) via fluent-bit and would benefit from being able to set higher compression levels for zstd.

— Ivan Bushmarinov

---

Full disclosure: the note above is from the human submitter; the implementation and the rest of this description were written by an LLM under human direction and review (Anthropic Claude Fable 5, model id claude-fable-5, via Claude Code 2.1.226).

What this adds

A compression_level option on out_kinesis_streams. Today the plugin compresses each record at zstd's fastest setting — flb_zstd_compress() hardcodes ZSTD_compress(..., 1) — with no way to trade CPU for ratio. For high-volume streams of large JSON records, higher zstd levels meaningfully reduce Kinesis throughput costs.

Implementation

Three commits, each preserving existing behavior exactly when the option is unset:

1. zstd: add flb_zstd_compress_level() — level-parameterized variant, validated against ZSTD_minCLevel()/ZSTD_maxCLevel(); flb_zstd_compress() delegates with its historical level 1.
2. aws compress: support explicit compression level — flb_aws_compression_compress_level() / flb_aws_compression_b64_truncate_compress_level() with a FLB_AWS_COMPRESS_LEVEL_DEFAULT sentinel; the compression-options table gains an optional level-aware function pointer (populated for zstd; gzip/snappy fall back to their built-in defaults with a warning). Existing entry points delegate with the sentinel — no call-site changes needed in other plugins (out_kinesis_firehose, out_s3 are untouched and can adopt the option later).
3. out_kinesis_streams: add compression_level option — config map entry + validation (compression_level without compression is a configuration error), threaded through both the per-record and aggregation send paths.

Testing

• New internal test test_compression_zstd_level (roundtrip at an explicit level, sentinel-equals-default, oversized-level rejection, non-tunable-codec fallback); full flb-it-aws_compress suite passes locally (macOS arm64, Release).
• flb-plugin-out_kinesis_streams compiles clean.

Happy to extend the option to out_kinesis_firehose/out_s3 in this PR or a follow-up if preferred, and to add the docs-repo page update.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cda7c98c-6d5b-4ef0-ba10-1a81781ff87b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}