Skip to content

fix(monitoring): Migrate Grafana to grafana-community chart and fix PVC deadlock #927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ybezsonov merged 2 commits into from
Jul 3, 2026
Merged

fix(monitoring): Migrate Grafana to grafana-community chart and fix PVC deadlock #927

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be4fa93
fix(monitoring): Migrate Grafana Helm chart to grafana-community repo
Jul 3, 2026
f990da5
fix(monitoring): Use Recreate strategy for Grafana to avoid PVC deadlock
Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions infra/scripts/setup/monitoring.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ trap cleanup EXIT
# Setup
kubectl create namespace "$NAMESPACE" 2>/dev/null || true
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts || true
helm repo add grafana https://grafana.github.io/helm-charts || true
helm repo add grafana-community https://grafana-community.github.io/helm-charts || true
helm repo update

# Grafana secret
Expand Down Expand Up @@ -116,6 +116,13 @@ persistence:
storageClassName: gp3
size: 10Gi

# The PVC above is ReadWriteOnce (EBS). RollingUpdate (the chart default)
# starts the new pod before the old one is deleted, so the new pod cannot
# attach the volume still held by the old pod and the upgrade deadlocks.
# Recreate terminates the old pod first, releasing the volume.
deploymentStrategy:
type: Recreate

resources:
requests:
cpu: 100m
Expand Down Expand Up @@ -152,7 +159,7 @@ grafana.ini:
EOF

log_info "Deploying Grafana..."
helm upgrade --install grafana grafana/grafana \
helm upgrade --install grafana grafana-community/grafana \
--namespace "$NAMESPACE" \
--values "$GRAFANA_VALUES_FILE"

Expand Down
5 changes: 3 additions & 2 deletions infra/scripts/setup/perf-platform.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ kubectl get namespace "${NAMESPACE}" >/dev/null 2>&1 || {
}

helm repo add grafana https://grafana.github.io/helm-charts >/dev/null 2>&1 || true
helm repo add grafana-community https://grafana-community.github.io/helm-charts >/dev/null 2>&1 || true
helm repo update >/dev/null

CLUSTER_NAME="${PREFIX}-eks"
Expand Down Expand Up @@ -288,11 +289,11 @@ GRAFANA_URL="http://${GRAFANA_LB}"

# Install Profiles Drilldown plugin (idempotent).
log_info "Installing Grafana Profiles Drilldown plugin..."
helm upgrade --install grafana grafana/grafana \
helm upgrade --install grafana grafana-community/grafana \
--namespace "${NAMESPACE}" \
--reuse-values \
--set "plugins={grafana-pyroscope-app}" \
--wait --timeout 5m
--wait --timeout 10m
log_success "Profiles Drilldown plugin installed"

# Wait for Grafana API to be ready after the helm upgrade restarts the pod.
Expand Down