Iceberg stress testing and performance tuning#1599
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughThis PR reorganizes Iceberg documentation by extracting troubleshooting content and performance tuning guidance from Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
kbatuigas
changed the title
kbatuigas
force-pushed
the
DOC-1848-Document-feature-Iceberg-stress-testing
branch
from
0041b57 to
d9955d4
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
modules/manage/pages/iceberg/specify-iceberg-schema.adoc (1)
63-63: Use empty-bracket xrefs for consistency with repo linking style.Consider
xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[]andxref:manage:iceberg/iceberg-troubleshooting.adoc[]to avoid hard-coded labels.Based on learnings, prefer AsciiDoc xref links with empty brackets to auto-resolve link titles from target pages.
Also applies to: 187-187
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@modules/manage/pages/iceberg/specify-iceberg-schema.adoc` at line 63, Replace hard-coded AsciiDoc xref labels with empty-bracket xrefs so titles auto-resolve: change occurrences of "xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[dead-letter queue]" and "xref:manage:iceberg/iceberg-troubleshooting.adoc[Iceberg troubleshooting]" (or any other hard-coded label at the other occurrence) to "xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[]" and "xref:manage:iceberg/iceberg-troubleshooting.adoc[]" respectively to follow the repo linking style.modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc (1)
249-249: Prefer title-derived xrefs instead of hard-coded link text.Use
xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[]so link text stays aligned automatically if the target title changes.Based on learnings, prefer AsciiDoc xref links with empty brackets so the referenced document title is rendered automatically.
Also applies to: 296-296
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc` at line 249, Replace the hard-coded link text "dead-letter queue (DLQ) table" with an AsciiDoc title-derived xref by using xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[] so the link text is rendered from the target title; update the occurrence in the sentence mentioning the iceberg_dlq_table_suffix property and also make the same change at the other occurrence referenced (line ~296) to ensure both links use the empty-bracket xref form.modules/manage/pages/iceberg/iceberg-performance-tuning.adoc (1)
25-25: Consider normalizing xrefs toxref:...[]across the new page.Using title-derived xrefs improves maintainability and keeps link text in sync with target page titles.
Based on learnings, AsciiDoc links in this repo should prefer
xref:...[]so displayed text is pulled from the referenced document.Also applies to: 33-33, 77-79, 108-108, 137-146
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@modules/manage/pages/iceberg/iceberg-performance-tuning.adoc` at line 25, Replace any xref that hardcodes link text with a title-derived xref so AsciiDoc pulls the target page title; for example change xref:manage:iceberg/about-iceberg-topics.adoc[About Iceberg Topics] to xref:manage:iceberg/about-iceberg-topics.adoc[] and apply the same change to all other explicit xref[...] instances on the iceberg-performance-tuning.adoc page so displayed text is sourced from the referenced document.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@modules/manage/pages/iceberg/iceberg-performance-tuning.adoc`:
- Line 25: Replace any xref that hardcodes link text with a title-derived xref
so AsciiDoc pulls the target page title; for example change
xref:manage:iceberg/about-iceberg-topics.adoc[About Iceberg Topics] to
xref:manage:iceberg/about-iceberg-topics.adoc[] and apply the same change to all
other explicit xref[...] instances on the iceberg-performance-tuning.adoc page
so displayed text is sourced from the referenced document.
In `@modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc`:
- Line 249: Replace the hard-coded link text "dead-letter queue (DLQ) table"
with an AsciiDoc title-derived xref by using
xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[] so the link
text is rendered from the target title; update the occurrence in the sentence
mentioning the iceberg_dlq_table_suffix property and also make the same change
at the other occurrence referenced (line ~296) to ensure both links use the
empty-bracket xref form.
In `@modules/manage/pages/iceberg/specify-iceberg-schema.adoc`:
- Line 63: Replace hard-coded AsciiDoc xref labels with empty-bracket xrefs so
titles auto-resolve: change occurrences of
"xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[dead-letter
queue]" and "xref:manage:iceberg/iceberg-troubleshooting.adoc[Iceberg
troubleshooting]" (or any other hard-coded label at the other occurrence) to
"xref:manage:iceberg/iceberg-troubleshooting.adoc#dead-letter-queue[]" and
"xref:manage:iceberg/iceberg-troubleshooting.adoc[]" respectively to follow the
repo linking style.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 10ce3270-f91a-40f6-a356-52466a149ef5
📒 Files selected for processing (6)
docs-data/property-overrides.jsonmodules/manage/pages/iceberg/about-iceberg-topics.adocmodules/manage/pages/iceberg/iceberg-performance-tuning.adocmodules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adocmodules/manage/pages/iceberg/iceberg-troubleshooting.adocmodules/manage/pages/iceberg/specify-iceberg-schema.adoc
| To check the current values of key translation cluster properties: | ||
|
|
||
| [,bash] | ||
| ---- | ||
| rpk cluster config get datalake_translator_flush_bytes | ||
| rpk cluster config get iceberg_target_lag_ms | ||
| rpk cluster config get iceberg_target_backlog_size | ||
| ---- |
There was a problem hiding this comment.
Seems a bit odd to be in the section about monitoring?
Also might be worth mentioning how to get the topic properties, if we want to keep this somewhere?
There was a problem hiding this comment.
Removed cluster properties
| @@ -0,0 +1,118 @@ | |||
| = Troubleshoot Iceberg Topics | |||
There was a problem hiding this comment.
Just noting that there's also an admin endpoint for validating REST catalog connectivity: redpanda-data/redpanda#29677
There was a problem hiding this comment.
@andrwng I see that this is under internal/, will this be exposed to end users? Right now we don't have any of these endpoints published in Admin v1 or v2
| If query latency is a concern and your workload produces large messages, consider: | ||
|
|
||
| * Reducing individual message sizes if your data model allows it. | ||
| * Increasing `datalake_translator_flush_bytes` to produce Parquet files with more records per file. |
There was a problem hiding this comment.
The key cluster config properties they'll want to change are:
datalake_translator_flush_bytes- 32MiB defaulticeberg_target_lag_ms- 1min default
Currently we'll translate records on a given partition for 30s. Afterwards we'll check if either the total translated size exceeds datalake_translator_flush_bytes or if the oldest translate record exceeds iceberg_target_lag_ms. If either are true we upload the parquet file.
This can mean that parquet files can be larger than 32MiB if the average message rate is 2 or more per 30s on a given partition. Even with the default cluster config values for datalake_translator_flush_bytes and iceberg_target_lag_ms.
However, this is a subtlety with our implementation and end users should increase datalake_translator_flush_bytes and iceberg_target_lag_ms so that they get a decent number of records per parquet file. A very rough est could be:
messages_per_file = min(
datalake_translator_flush_bytes / message_size,
throughput * iceberg_target_lag_ms
)
All of this to say that we might want to include iceberg_target_lag_ms in the docs here lol.
There was a problem hiding this comment.
Included iceberg_target_lag_ms
| ); | ||
| ---- | ||
|
|
||
| [,bash,role=no-copy] |
There was a problem hiding this comment.
Sometimes these tables look nicer when you add a no-wrap role to the code block.
JakeSCahill
left a comment
JakeSCahill
left a comment
There was a problem hiding this comment.
Nice IA improvements
|
@kbatuigas reminder to update cloud-docs & nav.adoc |
| branches: [v/*, shared, site-search,'!v-end-of-life/*'] | ||
| - url: https://github.com/redpanda-data/cloud-docs | ||
| branches: 'main' | ||
| branches: 'DOC-1848-iceberg-troubleshooting' |
There was a problem hiding this comment.
| branches: 'DOC-1848-iceberg-troubleshooting' | |
| branches: 'main' |
5 participants
Description
Related PR to add docs in Cloud: redpanda-data/cloud-docs#565
This pull request introduces significant improvements to the Iceberg documentation by splitting out performance tuning and troubleshooting content into dedicated pages, enhancing navigation, and updating cross-references for clarity. The changes make it easier for users to find best practices for optimizing and troubleshooting Iceberg topics, and ensure that documentation links are accurate and up-to-date.
Documentation Restructuring and Navigation:
nav.adoc, improving discoverability of these topics.New and Updated Content:
Created a new
iceberg-performance-tuning.adocpage with comprehensive guidance on partitioning, compaction, flush threshold and lag tuning, cluster sizing, and monitoring translation performance for Iceberg topics.Created a new standalone
iceberg-troubleshooting.adocpage that hosts content currently at About Iceberg Topics > Troubleshoot errors. Added list of relevant metrics.Resolves https://redpandadata.atlassian.net/browse/
Review deadline:
Page previews
Tune Performance for Iceberg Topics
Troubleshoot Iceberg Topics
Checks