fix(sentry): pin clickhouse to the version sentry 26.7 actually needs - #158
Merged
Conversation
The install never finished. sentry-snuba-migrate failed seven times with
MigrationInProgress: events_analytics_platform:
0050_add_attributes_array_column
and the migrations table showed 0049 and 0050 both stuck at in_progress: a
first run died mid-migration, and every later run refuses to touch a group
that still has one open.
The cause is the ClickHouse version. The sentry-kubernetes clickhouse chart
carries appVersion 23.8, while upstream self-hosted Sentry builds its image
from altinity/clickhouse-server:25.3.6.10034.altinitystable. Choosing that
chart was justified in the design as "the version Snuba is tested against" --
true once, but not for Sentry 26.7, whose events_analytics_platform
migrations need the newer server.
Also disables the replica StatefulSet. It is gated on
remote_servers.replica.backup.enabled, not on replicas, so `replicas: 1` left
a second unsynchronisable copy running -- there is no Zookeeper for it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Qx7mKDeDhysBWsqtqM1yr
TheMeinerLP
added a commit
that referenced
this pull request
Aug 16, 2026
…ers dir (#162) The image pin from #158 was necessary but not sufficient: the upgrade still failed and Flux rolled back to 23.8 five times, reporting StatefulSet/sentry/clickhouse status: 'Failed' The image itself is fine -- it pulls and runs. What breaks is the entrypoint. ClickHouse images from 24.x on run a user-setup step: with neither CLICKHOUSE_USER nor CLICKHOUSE_PASSWORD set, it writes users.d/default-user.xml to disable network access for `default`. This chart mounts users.d from a ConfigMap, so that path is read-only and the container dies before the server starts: /entrypoint.sh: line 156: /etc/clickhouse-server/users.d/default-user.xml: Read-only file system Verified by running the image with the chart's three ConfigMaps mounted: it fails exactly there, and starts cleanly once CLICKHOUSE_SKIP_USER_SETUP=1 is set. The chart's users ConfigMap already defines `default`, so skipping the step is correct rather than a workaround. The chart has no env: block -- it only exports SHARD in the container command -- so this cannot come from values and is patched in via the postRenderer that already sets the priority class. Claude-Session: https://claude.ai/code/session_017Qx7mKDeDhysBWsqtqM1yr Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
TheMeinerLP
added a commit
that referenced
this pull request
Aug 16, 2026
…ers dir (#164) The image pin from #158 was necessary but not sufficient: the upgrade still failed and Flux rolled back to 23.8 five times, reporting StatefulSet/sentry/clickhouse status: 'Failed' The image itself is fine -- it pulls and runs. What breaks is the entrypoint. ClickHouse images from 24.x on run a user-setup step: with neither CLICKHOUSE_USER nor CLICKHOUSE_PASSWORD set, it writes users.d/default-user.xml to disable network access for `default`. This chart mounts users.d from a ConfigMap, so that path is read-only and the container dies before the server starts: /entrypoint.sh: line 156: /etc/clickhouse-server/users.d/default-user.xml: Read-only file system Verified by running the image with the chart's three ConfigMaps mounted: it fails exactly there, and starts cleanly once CLICKHOUSE_SKIP_USER_SETUP=1 is set. The chart's users ConfigMap already defines `default`, so skipping the step is correct rather than a workaround. The chart has no env: block -- it only exports SHARD in the container command -- so this cannot come from values and is patched in via the postRenderer that already sets the priority class. Claude-Session: https://claude.ai/code/session_017Qx7mKDeDhysBWsqtqM1yr Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the failed install from #156 and corrects a wrong call in the design.
Symptom
sentry-snuba-migratefailed seven times, so the Sentry HelmRelease never went Ready:Migrations 0049 and 0050 are wedged at
in_progress: a first run died mid-migration, and every later run refuses to touch a group that still has one open. ClickHouse itself is healthy — zero restarts, no OOM.Cause
The
clickhousechart from sentry-kubernetes carries appVersion 23.8, while upstream self-hosted Sentry builds fromaltinity/clickhouse-server:25.3.6.10034.altinitystable.The design justified that chart as "the version Snuba is tested against". True once; not for Sentry 26.7, whose
events_analytics_platformmigrations need the newer server. My mistake.Fix
image/imageVersionto the Altinity build upstream uses.remote_servers.replica.backup.enabled, not onreplicas, soreplicas: 1still leftclickhouse-replica-0running — with no Zookeeper it can never sync.Rendered: one StatefulSet, correct image,
replicas: 1, no replica services.Manual step after merge
ClickHouse holds no real data yet, so the wedged state is cleared by starting fresh:
🤖 Generated with Claude Code