out_arvancloud_cloudlogs: add new output plugin for ArvanCloud CloudLogs#11095
out_arvancloud_cloudlogs: add new output plugin for ArvanCloud CloudLogs#11095javadnasrolahi wants to merge 10 commits into
Conversation
javadnasrolahi
requested review from
celalettin1286,
edsiper,
fujimotos,
koleini,
niedbalski and
patrick-stephens
as code owners
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an ArvanCloud CloudLogs Fluent Bit output plugin: public header and config creation/destruction, timestamp helpers and formatter, HTTP flush with optional gzip/auth, lifecycle callbacks, config map and plugin descriptor. Build flags and plugin registration are also added. ChangesArvanCloud CloudLogs plugin
Sequence Diagram(s)sequenceDiagram
participant FB as Fluent Bit Core
participant Init as cb_arvancloud_init
participant Cfg as flb_arvancloud_conf_create
participant Ctx as Plugin Context
participant Flush as cb_arvancloud_flush
participant Format as arvancloud_format
participant HTTP as Fluent Bit HTTP Client
participant AC as ArvanCloud API
FB->>Init: init plugin
Init->>Cfg: create & validate config (apikey, keys)
Cfg->>Ctx: allocate context, setup upstream (https, host, port, uri)
Ctx-->>FB: return context
loop For each event chunk
FB->>Flush: flush events
Flush->>Format: decode events, build JSON logs array
Format->>Format: resolve timestamps & logType per record
Format-->>Flush: formatted payload
alt gzip enabled
Flush->>Flush: gzip compress payload
end
Flush->>HTTP: POST payload (headers + optional Authorization)
HTTP->>AC: send request
AC-->>HTTP: respond (status)
HTTP-->>Flush: deliver status
alt 2xx
Flush-->>FB: FLB_OK
else retryable (5xx / network)
Flush-->>FB: FLB_RETRY
else client error (4xx)
Flush-->>FB: FLB_ERROR
end
end
FB->>Init: shutdown
Init->>Ctx: flb_arvancloud_conf_destroy -> free resources
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
CMakeLists.txt(1 hunks)cmake/plugins_options.cmake(1 hunks)plugins/CMakeLists.txt(1 hunks)plugins/out_arvancloud_cloudlogs/CMakeLists.txt(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.
Applied to files:
cmake/plugins_options.cmakeCMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.
Applied to files:
cmake/plugins_options.cmakeCMakeLists.txt
🧬 Code graph analysis (3)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-156)flb_arvancloud_conf_destroy(158-189)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (7)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)include/fluent-bit/flb_output.h (1)
flb_output_config_map_set(1298-1321)src/flb_output.c (2)
flb_output_get_property(1096-1099)flb_output_upstream_set(1556-1647)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)src/flb_utils.c (1)
flb_utils_url_split(1441-1534)src/flb_record_accessor.c (2)
flb_ra_create(271-358)flb_ra_destroy(232-248)src/flb_upstream.c (2)
flb_upstream_create(290-363)flb_upstream_destroy(656-698)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (11)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-156)flb_arvancloud_conf_destroy(158-189)
This patch adds a new output plugin that sends logs to ArvanCloud CloudLogs service. The plugin supports: - GZIP compression - Configurable timestamp formats - Custom log fields mapping - Retry logic and error handling The plugin has been tested with ArvanCloud CloudLogs API and successfully delivers logs with proper formatting and timestamps. Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
javadnasrolahi
force-pushed
the
add-out-arvancloud-cloudlogs
branch
from
c414de0 to
b14d48d
Compare
patrick-stephens
left a comment
patrick-stephens
left a comment
There was a problem hiding this comment.
Please follow the PR template and ensure everything it requires is covered - in particular support for all targets.
Please respond to all comments raised by Coderabbit in some way to indicate whether it is relevant or not.
| DEFINE_OPTION(FLB_OUT_UDP "Enable UDP output plugin" ON) | ||
| DEFINE_OPTION(FLB_OUT_VIVO_EXPORTER "Enable Vivo exporter output plugin" ON) | ||
| DEFINE_OPTION(FLB_OUT_WEBSOCKET "Enable Websocket output plugin" ON) | ||
| DEFINE_OPTION(FLB_OUT_ARVANCLOUD_CLOUDLOGS "Enable ArvanCloud CloudLogs output plugin" ON) |
There was a problem hiding this comment.
Will this build for all Linux, macOS and Windows targets with no extra dependencies required?
If it will not build for Windows or macOS then configure the defaults under the cmake/ directory for those targets.
If it needs more build or runtime dependencies then please update the container definitions for both the OSS images under dockerfiles/ and also the package builds under packaging/distros/
There was a problem hiding this comment.
yes, This plugin will build for all Linux, macOS, and Windows targets with no extra dependencies required.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (1)
158-189: Memory leak still present: config-map-populated fields not freed.As previously noted,
flb_output_config_map_set()allocates SDS strings for configuration fields likeapi_key,log_type,log_type_key,tag_key,timestamp_key, andtimestamp_format. These are not being freed in the destructor, causing memory leaks on every plugin teardown.Apply the previously suggested diff to add cleanup for these fields:
if (ctx->proxy) { flb_sds_destroy(ctx->proxy); } + if (ctx->api_key) { + flb_sds_destroy(ctx->api_key); + } if (ctx->proxy_host) { flb_free(ctx->proxy_host); } if (ctx->scheme) { flb_sds_destroy(ctx->scheme); } if (ctx->host) { flb_sds_destroy(ctx->host); } if (ctx->uri) { flb_sds_destroy(ctx->uri); } + if (ctx->log_type) { + flb_sds_destroy(ctx->log_type); + } + if (ctx->log_type_key) { + flb_sds_destroy(ctx->log_type_key); + } if (ctx->ra_log_type_key) { flb_ra_destroy(ctx->ra_log_type_key); } + if (ctx->tag_key) { + flb_sds_destroy(ctx->tag_key); + } + if (ctx->timestamp_key) { + flb_sds_destroy(ctx->timestamp_key); + } if (ctx->ra_timestamp_key) { flb_ra_destroy(ctx->ra_timestamp_key); } + if (ctx->timestamp_format) { + flb_sds_destroy(ctx->timestamp_format); + }
🧹 Nitpick comments (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (1)
129-131: Simplify:flb_sds_destroyhandles NULL safely.The explicit NULL check is redundant since
flb_sds_destroy()already handles NULL pointers gracefully.Apply this diff:
- if (ctx->uri) { - flb_sds_destroy(ctx->uri); - } + flb_sds_destroy(ctx->uri);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (7)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)include/fluent-bit/flb_output.h (1)
flb_output_config_map_set(1298-1321)src/flb_output.c (2)
flb_output_get_property(1096-1099)flb_output_upstream_set(1556-1647)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)src/flb_utils.c (1)
flb_utils_url_split(1441-1534)src/flb_record_accessor.c (2)
flb_ra_create(271-358)flb_ra_destroy(232-248)src/flb_upstream.c (2)
flb_upstream_create(290-363)flb_upstream_destroy(656-698)
🔇 Additional comments (5)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (5)
53-57: Past issue resolved: proper cleanup on config map failure.The error path now correctly calls the destructor to clean up any partially-initialized fields from
flb_output_config_map_set().
59-78: LGTM: proxy parsing and memory management.The proxy URL parsing correctly transfers ownership of the host string to
ctx->proxy_hostand frees the other components.
80-93: LGTM: HTTPS enforcement and required field validation.Forcing TLS and validating the required API key early prevents configuration errors from propagating.
95-115: LGTM: conditional record accessor initialization.The record accessors are correctly initialized only when configured, with clear error messages and proper cleanup.
138-156: LGTM: upstream creation and configuration.The upstream is correctly created with either proxy or direct connection settings, and properly configured with TLS flags.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (1)
419-422: Proxy configuration never propagated to HTTP client.When
proxyis configured, the HTTP client receivesNULLfor both host and proxy parameters. This causes incorrectHostheaders and prevents CONNECT tunneling, so proxied requests fail. Pass the destination host/port and proxy string so the client can negotiate the tunnel correctly.Apply this diff:
c = flb_http_client(u_conn, FLB_HTTP_POST, ctx->uri, final_payload, final_payload_size, - NULL, 0, // host=NULL, port=0 -> use from u_conn - NULL, 0); // proxy=NULL + ctx->host, ctx->port, + ctx->proxy, 0);plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (1)
159-189: Past review concern remains: config-map SDS fields still leak.The destructor still doesn't free SDS strings populated by
flb_output_config_map_set()on lines 53. Fieldsapi_key,log_type,log_type_key,tag_key,timestamp_key, andtimestamp_formatare allocated as SDS by the config map but never destroyed here, leaking memory on every teardown.Apply this diff to plug the leaks:
if (ctx->proxy) { flb_sds_destroy(ctx->proxy); } + if (ctx->api_key) { + flb_sds_destroy(ctx->api_key); + } if (ctx->proxy_host) { flb_free(ctx->proxy_host); } if (ctx->scheme) { flb_sds_destroy(ctx->scheme); } if (ctx->host) { flb_sds_destroy(ctx->host); } if (ctx->uri) { flb_sds_destroy(ctx->uri); } + if (ctx->log_type) { + flb_sds_destroy(ctx->log_type); + } + if (ctx->log_type_key) { + flb_sds_destroy(ctx->log_type_key); + } if (ctx->ra_log_type_key) { flb_ra_destroy(ctx->ra_log_type_key); } + if (ctx->tag_key) { + flb_sds_destroy(ctx->tag_key); + } + if (ctx->timestamp_key) { + flb_sds_destroy(ctx->timestamp_key); + } if (ctx->ra_timestamp_key) { flb_ra_destroy(ctx->ra_timestamp_key); } + if (ctx->timestamp_format) { + flb_sds_destroy(ctx->timestamp_format); + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (7)
include/fluent-bit/flb_mem.h (2)
flb_calloc(84-96)flb_free(126-128)include/fluent-bit/flb_output.h (1)
flb_output_config_map_set(1298-1321)src/flb_output.c (2)
flb_output_get_property(1096-1099)flb_output_upstream_set(1556-1647)src/flb_sds.c (2)
flb_sds_create(78-90)flb_sds_destroy(389-399)src/flb_utils.c (1)
flb_utils_url_split(1441-1534)src/flb_record_accessor.c (2)
flb_ra_create(271-358)flb_ra_destroy(232-248)src/flb_upstream.c (2)
flb_upstream_create(290-363)flb_upstream_destroy(656-698)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (12)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_mp.c (1)
flb_mp_count(43-46)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-157)flb_arvancloud_conf_destroy(159-190)
🔇 Additional comments (5)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (5)
60-134: LGTM: Robust timestamp parsing with auto-detection.The function properly validates inputs, attempts user-provided format or auto-detects ISO8601 patterns, and correctly extracts fractional seconds with nanosecond precision scaling.
136-351: Well-structured formatting logic with proper fallbacks.The function correctly:
- Initializes and cleans up the decoder
- Implements priority-based log_type extraction (record field → configured default)
- Handles timezone-aware timestamp parsing with awareness of
timegm()side effects (Line 276 comment)- Falls back gracefully to event timestamps on parse failures
- Properly cleans up intermediate SDS allocations
462-523: Appropriate HTTP response handling with retry logic.The status code mapping is correct:
- 2xx → success
- 400 → error (client fault, no retry)
- 401/403 → error (auth failure, no retry)
- 429 → retry (rate limit)
- 5xx → retry (server fault)
- Others → error
536-555: LGTM: Init and exit callbacks properly delegate to config lifecycle functions.The callbacks correctly create and destroy the plugin context with appropriate error handling.
558-622: LGTM: Config map and plugin registration are well-defined.The configuration options are clearly documented with appropriate defaults, and the plugin struct correctly declares network output with TLS support. Including
test_formatter.callbackenables testing of the formatting logic.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (1)
41-53: Add error handling forstrftimeandsnprintfto prevent buffer overruns.The missing validation of
strftime's return value and buffer space checks before callingsnprintfcan lead to buffer overruns if the buffer is too small or ifstrftimefails. This issue was previously flagged and remains unaddressed.Apply the previously suggested fix:
static size_t format_timestamp_rfc3339(char *buffer, size_t buffer_size, time_t seconds, long nsec) { struct tm tm; size_t s; int len; gmtime_r(&seconds, &tm); s = strftime(buffer, buffer_size - 1, "%Y-%m-%dT%H:%M:%S", &tm); + if (s == 0 || s >= buffer_size - 1) { + return 0; + } len = snprintf(buffer + s, buffer_size - 1 - s, ".%06ldZ", (long) (nsec / 1000)); + if (len < 0 || (size_t)len >= buffer_size - 1 - s) { + return 0; + } return s + len; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (13)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_mp.c (1)
flb_mp_count(43-46)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-157)flb_arvancloud_conf_destroy(159-190)src/flb_output.c (1)
flb_output_set_context(1524-1527)
🔇 Additional comments (2)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (2)
246-306: Well-implemented timestamp extraction with proper timezone handling.The timestamp extraction logic correctly handles timezone-aware timestamps by reading
gmtoffbefore callingtimegm()(which may modify the struct) and applying the offset to convert to UTC. The fallback to event timestamp with a warning when parsing fails is appropriate.
466-534: Appropriate HTTP response handling with correct retry logic.The HTTP status code handling correctly distinguishes between retryable errors (429, 5xx) and non-retryable errors (4xx except 429), with helpful logging for debugging. This follows best practices for resilient HTTP clients.
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (12)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_mp.c (1)
flb_mp_count(43-46)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-157)flb_arvancloud_conf_destroy(159-190)
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (1)
41-53: Consider adding error handling for strftime and snprintf.The function doesn't validate the return values from
strftimeandsnprintf, which could lead to issues if the buffer is too small or if formatting fails. While the current usage with a 64-byte buffer is safe, adding validation would improve robustness.Apply this diff to add error handling:
static size_t format_timestamp_rfc3339(char *buffer, size_t buffer_size, time_t seconds, long nsec) { struct tm tm; size_t s; int len; + if (buffer_size < 32) { /* Minimum required for RFC3339 */ + return 0; + } + gmtime_r(&seconds, &tm); s = strftime(buffer, buffer_size - 1, "%Y-%m-%dT%H:%M:%S", &tm); + if (s == 0 || s >= buffer_size - 1) { + return 0; + } + len = snprintf(buffer + s, buffer_size - 1 - s, ".%06ldZ", (long) (nsec / 1000)); + if (len < 0 || (size_t)len >= buffer_size - 1 - s) { + return 0; + } + return s + len; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-157)flb_arvancloud_conf_destroy(159-190)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (12)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_mp.c (1)
flb_mp_count(43-46)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-157)flb_arvancloud_conf_destroy(159-190)
🔇 Additional comments (6)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h (1)
20-62: LGTM! Clean header design.The header is well-structured with a clear separation between network configuration, plugin configuration, and instance fields. The public API declarations align with the implementation in the configuration file.
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (5)
62-117: LGTM! Robust timestamp parsing.The function properly validates inputs, initializes structures, handles fractional seconds extraction, and includes appropriate error checking. The timezone offset handling was correctly addressed based on previous review feedback.
119-343: LGTM! Comprehensive formatting logic.The formatting function is well-structured with proper error handling:
- Decoder initialization checks for errors
- Log type extraction follows the correct priority order (log_type_key > log_type)
- Timestamp handling gracefully falls back from custom format → raw value → event timestamp
- Proper cleanup of allocated resources (log_type_value, decoder)
- Timezone offset is correctly read before calling timegm
345-534: LGTM! Robust flush implementation.The flush callback is well-implemented with:
- Proper resource acquisition and cleanup (upstream connection, payloads, HTTP client)
- Correct proxy parameter handling (addressed from previous reviews)
- Graceful fallback when gzip compression fails
- Appropriate HTTP status code handling (retry for rate limits and server errors, error for auth/client failures)
- Consistent cleanup in all code paths
536-555: LGTM! Standard lifecycle callbacks.The init and exit callbacks follow the standard Fluent Bit plugin pattern, properly delegating to the configuration management functions and handling errors appropriately.
557-616: LGTM! Complete configuration and plugin registration.The configuration map provides clear documentation for all options with appropriate defaults and constraints. The plugin registration properly declares all callbacks, the test formatter, and network/TLS flags suitable for an HTTPS output plugin.
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (1)
41-53: Harden RFC3339 formatter againststrftime/snprintffailures and buffer edge cases
format_timestamp_rfc3339()assumesstrftimeandsnprintfalways succeed and never hit the buffer limit; on failure,smay be 0 andlennegative or ≥ remaining space, and the returned length can be inconsistent with the actual string content. The current usage with a 64‑byte buffer is practically safe, but tightening this helper avoids surprises if it’s reused.You can guard both calls and bail out on error:
static size_t format_timestamp_rfc3339(char *buffer, size_t buffer_size, time_t seconds, long nsec) { struct tm tm; size_t s; int len; - + gmtime_r(&seconds, &tm); - s = strftime(buffer, buffer_size - 1, "%Y-%m-%dT%H:%M:%S", &tm); - len = snprintf(buffer + s, buffer_size - 1 - s, - ".%06ldZ", (long) (nsec / 1000)); - return s + len; + s = strftime(buffer, buffer_size - 1, "%Y-%m-%dT%H:%M:%S", &tm); + if (s == 0 || s >= buffer_size - 1) { + return 0; + } + + len = snprintf(buffer + s, buffer_size - 1 - s, + ".%06ldZ", (long) (nsec / 1000)); + if (len < 0 || (size_t) len >= buffer_size - 1 - s) { + return 0; + } + + return s + len; }Callers should treat a 0 return as “failed to format” and fall back appropriately.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h(1 hunks)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.h
- plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c
🧰 Additional context used
🧬 Code graph analysis (1)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (12)
include/fluent-bit/flb_compat.h (1)
gmtime_r(75-81)src/flb_strptime.c (1)
flb_strptime(247-251)src/flb_mp.c (1)
flb_mp_count(43-46)src/flb_log_event_decoder.c (3)
flb_log_event_decoder_init(103-120)flb_log_event_decoder_next(309-456)flb_log_event_decoder_destroy(151-178)src/flb_record_accessor.c (1)
flb_ra_translate(628-633)src/flb_sds.c (3)
flb_sds_destroy(389-399)flb_sds_create_size(92-95)flb_sds_cat(120-141)src/flb_pack.c (1)
flb_msgpack_raw_to_json_sds(1026-1085)src/flb_upstream.c (2)
flb_upstream_conn_get(711-844)flb_upstream_conn_release(862-947)src/flb_gzip.c (1)
flb_gzip_compress(157-252)src/flb_http_client.c (5)
flb_http_client(814-859)flb_http_add_header(963-995)flb_http_set_content_encoding_gzip(1123-1132)flb_http_do(1572-1632)flb_http_client_destroy(1688-1695)include/fluent-bit/flb_mem.h (1)
flb_free(126-128)plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c (2)
flb_arvancloud_conf_create(27-122)flb_arvancloud_conf_destroy(124-149)
🔇 Additional comments (4)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.c (4)
62-117: Timestamp parsing helper looks consistent and focused
parse_timestamp_string()cleanly validates inputs, zero‑initializesstruct flb_tm, delegates parsing toflb_strptime, and separately derives fractional seconds up to nanosecond precision. The behavior matches the comment (“no auto‑detection; format must match input”) and integrates correctly with the later UTC normalization logic.
119-343: Formatter logic and resource handling look solid overallThe core formatter correctly:
- Uses
event_chunk->total_events(orflb_mp_countin test mode) for thelogsarray size.- Iterates with
flb_log_event_decoder, honoring normal log events and destroying the decoder at the end.- Derives
logTypewith the intended precedence (log_type_keyvia record accessor, thenlog_type), freeing any temporary SDS.- Normalizes timestamps to RFC3339 UTC when
timestamp_formatis provided, otherwise falling back to the event timestamp or the original field as configured.- Wraps the original record as
payloadand conditionally includes the tag key.Memory ownership of the decoder, MessagePack buffer, and temporary SDS values is handled cleanly apart from the
timestamp_valueleak noted separately.
345-526: Flush path: connection lifecycle and payload handling look correct
cb_arvancloud_flush():
- Safely obtains/releases the upstream connection and retries on connection failure.
- Delegates formatting to
arvancloud_formatand correctly frees the JSON SDS in both success and error paths.- Optionally compresses with gzip, tracking whether an extra heap buffer was allocated and freeing it conditionally.
- Builds a single HTTP POST with the expected headers (User‑Agent, Content‑Type, optional
Authorization: apikey …, optionalContent-Encoding: gzip).- Maps
flb_http_do()/HTTP status codes toFLB_OK,FLB_RETRY, orFLB_ERRORwith sensible logging.- Always destroys the HTTP client, releases the upstream connection, and frees any allocated payload buffers before returning.
No functional or resource‑management issues stand out here.
599-608: Plugin registration and flags look appropriateThe plugin descriptor wires init/flush/exit and
test_formattercorrectly, andFLB_OUTPUT_NET | FLB_IO_TLSmatches the hard‑coded HTTPS upstream setup shown in the configuration helper. Nothing to change here.
|
This is a sample configuration showing how to use this plugin: |
niedbalski
left a comment
niedbalski
left a comment
There was a problem hiding this comment.
I just need a bit more information to move forward.
Could you clarify which service this refers to, and where the API specification can be found?
Also, is this component open-source, and who would be responsible for maintaining this particular plugin over time?
Thank you!
This plugin supports ArvanCloud CloudLogs, which is a large-scale log management solution similar to AWS CloudWatch or Google Cloud Logging. You can find more details on the product landing page: |
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
Still working on this. Could you let us know what exactly needs to be changed so we can get this merged as soon as possible? |
…rvancloud_conf_destroy in ArvanCloud CloudLogs plugin configuration Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
…r enhanced functionality Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
…n configuration Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
…pport proxy configurations Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
…Logs plugin Signed-off-by: Javad Nasrolahi <javadnasrolahi13@gmail.com>
javadnasrolahi
force-pushed
the
add-out-arvancloud-cloudlogs
branch
from
42dcca6 to
2337cd5
Compare
|
Please update the description to follow the template. Can you confirm it builds for macOS and Windows as well as all the Linux platforms? We will need some unit tests as well. |
I've updated the PR description to follow the template. The build works on macOS and Windows in addition to the Linux platforms. |
I added unit test. |
4 participants
Summary
This PR adds a new output plugin
arvancloud_cloudlogsfor sending logs to the ArvanCloud CloudLogs service.The plugin posts log events as JSON to the CloudLogs ingestion endpoint
(
https://napi.arvancloud.ir/logging/v1/entries/write) over HTTPS, wrappingeach Fluent Bit record into the CloudLogs request schema:
{ "logs": [ { "logType": "...", "timestamp": "YYYY-MM-DDTHH:MM:SS.ffffffZ", "severity": "INFO", "resource": { "type": "general" }, "payload": { ...original record... } } ] }Features
apikeyconfig option (sent asAuthorization: apikey <value>).napi.arvancloud.ir:443) with TLS enforcedthrough
FLB_IO_TLS.log_type(default:fluentbit), orlog_type_key(record accessor), whichtakes priority over
log_typewhen the field is present and non-empty.timestamp_key(record accessor) witha user-provided
timestamp_format(strptime-style). Parsed values arenormalized to RFC3339 UTC with microseconds. Falls back to the Fluent Bit
event timestamp when extraction or parsing fails. If
timestamp_formatisnot set, the field value is forwarded as-is.
gzip(setsContent-Encoding: gzipwhen enabled; gracefully falls back touncompressed on failure).
include_tag_key/tag_key.2xx→ success400,401,403→FLB_ERROR(no retry)429,5xx→FLB_RETRYFLB_ERRORFiles added / changed
plugins/out_arvancloud_cloudlogs/CMakeLists.txt– plugin buildplugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs.{c,h}– pluginimplementation (init, format, flush, exit, config map)
plugins/out_arvancloud_cloudlogs/arvancloud_cloudlogs_conf.c– contextcreation/destruction, record accessor setup, upstream setup
CMakeLists.txt,cmake/plugins_options.cmake,plugins/CMakeLists.txt– register the new plugin and addFLB_OUT_ARVANCLOUD_CLOUDLOGSoption (enabled by default and underFLB_ALL).Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
[N/A]Run local packaging test showing all targets (including any new ones) build.[N/A]Setok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
[N/A]Backport to latest stable release.Example configuration
Minimal configuration (only
apikeyis required):[OUTPUT] Name arvancloud_cloudlogs Match * apikey <your-api-key>Full configuration showing all supported options:
[OUTPUT] Name arvancloud_cloudlogs Match * apikey <your-api-key> log_type my-app log_type_key $log_type timestamp_key $time timestamp_format %Y-%m-%dT%H:%M:%S%z gzip true include_tag_key true tag_key tagFluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.