feat(cse): emit kubelet active settings as a structured JSON Kusto event

[abigailliang-aks-sig-node]

Summary

Emit kubelet's effective runtime configuration (CLI flags + config file content) as a structured JSON event in GuestAgentGenericLogs, queryable via parse_json() from Kusto.

Motivation

There is currently no easy way to query kubelet's active settings across nodes in Kusto. Debugging requires SSH-ing into individual nodes. This event enables a single Kusto query to surface kubelet configuration for all nodes in a cluster.

Design: one event per boot is sufficient

• Kubelet's startup flags/config are constant within a single boot — unlike memory usage, they don't drift, so there is no meaningful timechart to draw over a constant.
• Configuration only changes on restart/reconfig — and every restart re-emits the event. So "one event per boot" naturally covers every real configuration change point.

Architecture: async oneshot systemd service

The telemetry runs as an independent systemd oneshot service (emit-kubelet-active-flags.service), completely decoupled from the CSE critical path:

• WantedBy=kubelet.service, enabled at VHD build time — systemd automatically starts it whenever kubelet starts; no systemctl start call needed in cse_config.sh
• After=kubelet.service — guarantees /etc/default/kubelet and /etc/default/kubeletconfig.json are already written
• ConditionPathExists=!/run/emit-kubelet-active-flags.done + ExecStartPost=/bin/touch /run/emit-kubelet-active-flags.done — limits to one event per boot (prevents flooding if kubelet crashloops; /run is tmpfs, cleared on reboot)
• No CSE changes needed — cse_config.sh is not modified; the service is triggered entirely by systemd dependency ordering

This follows the same pattern as measure-tls-bootstrapping-latency.service.

Why not inline in CSE?

Even with & and FD redirection, a background subshell inherits the CSE pipe FDs. In scriptless phase 2, Go's cmd.Wait() blocks until all pipe write ends close — so the background process still delays CSE completion. A systemd oneshot service runs in its own scope with independent FDs, fully decoupled.

Why not a generic node-telemetry service?

This unit is deliberately kubelet-scoped rather than a shared "node config telemetry" service, because
the repo already establishes one unit per telemetry domain, each with a different trigger:

Unit	Trigger
cgroup-memory-telemetry	.timer — OnBootSec=0 + every 5 min
cgroup-pressure-telemetry	.timer — periodic
aks-check-network	After=network-online.target
emit-kubelet-active-flags	WantedBy=kubelet.service

Folding other domains (e.g. OS configuration: sysctl, THP, cgroup version, ulimits) into this unit would
be wrong on two counts:

• Trigger — OS settings have no kubelet dependency and are ready at boot. Gating them on
  kubelet.service means a node whose kubelet fails to start emits no OS telemetry, which is exactly
  when it is most needed for diagnosis.
• Size — Context1 is hard-truncated at 3072 bytes and this payload already drops config_file when it
  exceeds 3000. A second domain in the same Message would crowd out the kubelet data, so it has to be a
  separate event (separate TaskName) anyway.

Separate event + separate trigger implies a separate unit. What is worth sharing when a second consumer
appears is the event-writing boilerplate (the jq -n envelope, the size cap, the once-per-boot /run
marker) — currently duplicated across ~8 scripts in parts/linux/cloud-init/artifacts/. Note this script
is VHD-baked and intentionally does not source cse_helpers.sh, which is what keeps it off the CSE path.

Data sources: files, not journalctl

Reads directly from kubelet's config files on disk (instantaneous, no polling needed):

File	Contains
/etc/default/kubelet	KUBELET_FLAGS=--cloud-provider=external --max-pods=110 ... (command-line flags)
/etc/default/kubeletconfig.json	KubeletConfiguration JSON (39 translated flags: maxPods, clusterDNS, featureGates, etc.)

Both files are written by ensureKubelet() before kubelet starts, so they are guaranteed to exist when this service runs (After=kubelet.service).

Together these cover all RP-configurable kubelet settings: the 39 flags translated to config file + the command-line flags (notably --cloud-provider).

Size guard

GuestAgentGenericLogs.Context1 is hard-truncated at 3072 bytes (3 KiB), verified empirically (16M events, max Context1 length = 3072). We cap at 3000 bytes. If exceeded, config_file is dropped and replaced with "truncated:exceeded_size_cap", with a log warning.

Files

File	Purpose
parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh	Standalone script: reads config files, assembles JSON, writes guest agent event
parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.service	Systemd oneshot unit: After=kubelet.service, WantedBy=kubelet.service, once-per-boot guard
vhdbuilder/packer/packer_source.sh	Installs both files to VHD paths
vhdbuilder/packer/vhd-image-builder-*.json	Uploads both files during VHD bake
vhdbuilder/packer/pre-install-dependencies.sh	systemctl enable emit-kubelet-active-flags.service — creates the kubelet.service.wants/ symlink that makes WantedBy= take effect
vhdbuilder/packer/imagecustomizer/azlosguard/azlosguard.yml	Same install + enable for the OSGuard image

No changes to cse_config.sh or app.go — the service is fully self-contained.

Event payload

{
  "uses_config_file": true,
  "config_path": "/etc/default/kubeletconfig.json",
  "kubelet_flags_count": 5,
  "kubelet_flags": { "cloud-provider": "external", "node-ip": "10.224.0.5" },
  "config_file": { "maxPods": 110, "clusterDNS": ["10.0.0.10"] }
}

kubelet_flags comes from /etc/default/kubelet, config_file from /etc/default/kubeletconfig.json — so
kubelet_flags_count doubles as a migration-progress metric (how many settings have not moved into the config file yet).

Kusto query example

GuestAgentGenericLogs
| where TaskName == "AKS.CSE.ensureKubelet.kubeletActiveFlags"
| extend payload = parse_json(Context1)
| project PreciseTimeStamp, RoleInstanceName, Cluster,
    usesConfigFile = payload.uses_config_file,
    remainingFlags = payload.kubelet_flags_count,
    maxPods = payload.config_file.maxPods,
    clusterDNS = payload.config_file.clusterDNS,
    featureGates = payload.config_file.featureGates,
    cloudProvider = payload.kubelet_flags.["cloud-provider"],
    allFlags = payload.kubelet_flags

Test plan

• ShellSpec tests pass (819 examples, 0 failures)
• make generate produces no diff
• E2E via [Unofficial][Orchestration] AgentBaker + Linux VHD Full E2E Tests pipeline (bake + apply with current branch)

[abigailliang-aks-sig-node]

Tracking work item: AB#38936519

[github-actions]

Windows Unit Test Results

  3 files   12 suites   51s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit faff281.

♻️ This comment has been updated with latest results.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 173691175
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
First failing step: Test_AzureLinuxV3_MANA validating traffic through MANA VF via debug pod exec

Detective summary: VMSS/node provisioning, SSH, validation pod, MANA PCI, driver, and VF bonding all passed; the failure occurred when Kubernetes exec from the debug pod returned apiserver-to-kubelet proxy 502 Bad Gateway before the curl traffic check could prove a datapath issue.

Likely cause / signature: Transient kubelet streaming/exec transport failure; reusing existing wiki signature wireserver-exec-context-deadline-kubelet-streaming.
Wiki signature: wireserver-exec-context-deadline-kubelet-streaming

Confidence: High

Recommended owner/action: E2E/test-infra owner should track under the kubelet-streaming flake and consider retrying exec on 502/500/context-deadline validation failures; no PR-author action recommended from this signal alone.

Strongest alternative: Real MANA datapath regression, but less likely because MANA setup validations passed and the observed error is the Kubernetes exec proxy transport failing before curl results were available.

Evidence: timeline failed task exit 1; failed test run 545377067; task log 562 around line 799; build metadata and PR #9034 metadata.

[aks-node-assistant]

AgentBaker Linux gate detective

Run: 173784179
Failed job/stage/task: e2e / Run AgentBaker E2E / Run AgentBaker E2E
First failing step: Test_Ubuntu2204_ArtifactStreaming_ARM64 kernel-log validation

Detective summary: Ubuntu 22.04 ArtifactStreaming ARM64 provisioned successfully, reached VM validation, then kernel-log validation flagged an early boot Call trace/PANIC_CRASH after node Ready, SSH, validation pod, KSCR/bootstrap, and NIC/rx checks passed.

Likely cause / signature: Likely transient/platform kernel call-trace validation noise; the kubelet-active-flags JSON PR is unlikely causal because the failure is in kernel log scan after unrelated provisioning/validation succeeded.
Wiki signature: ubuntu2204-artifactstreaming-arm64-kernel-calltrace-validation

Confidence: Medium

Recommended owner/action: Node Lifecycle/E2E owners should track as a new Ubuntu2204 ArtifactStreaming ARM64 kernel-calltrace signature and inspect scenario kernel logs if it recurs.

Strongest alternative: A PR-induced kubelet/config output instability, but less likely because the PR surface is telemetry/config formatting and the node was otherwise healthy through validation.

Evidence: timeline failed task exit 1; failed test run 546207590; task log 562 around line 772; build/PR metadata.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:80

• The size guard only drops config_file once; if the payload is still > 3000 bytes (e.g., unusually long KUBELET_FLAGS values), GuestAgentGenericLogs.Context1 will still be truncated and parse_json() will fail. Consider re-checking size after dropping config_file, and if still oversized, also drop (or truncate) flags to guarantee the emitted payload remains valid JSON.

    if [ "${payload_bytes}" -gt "${max_message_bytes}" ]; then
        echo "kubelet active settings payload exceeded ${max_message_bytes} bytes (${payload_bytes}), dropping config_file content" >&2
        payload="$(jq -cn \
            --argjson uses_config_file "${uses_config_file}" \
            --arg config_path "${config_path}" \

[awesomenix]

This is oneshot service even if it crash loops, it will still be executed only once isnt it? since you also set Restart=no, may be you can get rid of it.

[abigailliang-aks-sig-node]

Type=oneshot + Restart=no means the service won't restart itself, but WantedBy=kubelet.service causes systemd to re-trigger it on every kubelet start (including crashloop restarts).

The fingerprint dedup in the script ensures we only emit when the config actually changes — so a crashlooping kubelet with unchanged config won't flood the events directory.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (3)

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:76

• The emitted payload uses field names flag_count and flags, but the PR description (event payload example + Kusto query) refers to kubelet_flags_count and kubelet_flags. This mismatch will make the documented query fail and makes the event schema ambiguous for consumers. Consider renaming the JSON fields in the emitted payload (and updating the ShellSpec accordingly), or update the PR description/schema documentation to match the implementation.

    payload="$(jq -cn \
        --argjson uses_config_file "${uses_config_file}" \
        --arg config_path "${config_path}" \
        --argjson flag_count "${flag_count}" \
        --argjson flags "${flags_json}" \
        --argjson config_file "${config_file_json}" \
        '{uses_config_file: $uses_config_file, config_path: $config_path, flag_count: $flag_count, flags: $flags, config_file: $config_file}')"

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.service:14

• The PR description specifies a once-per-boot guard using ConditionPathExists=!/run/emit-kubelet-active-flags.done plus an ExecStartPost touch, but the unit file currently has neither. Instead, the script deduplicates via a fingerprint file. Please reconcile the implementation with the described design (either add the /run done-marker guard in the unit, or update the design text to reflect fingerprint-based dedup and potential re-emits within a boot when config changes).

[Unit]
Description=Emit kubelet active flags telemetry event
After=kubelet.service
ConditionPathExists=/opt/azure/containers/emit-kubelet-active-flags.sh
# Reads /etc/default/kubelet and /etc/default/kubeletconfig.json (written by ensureKubelet
# before kubelet starts), so After=kubelet.service guarantees the files exist.
# The script dedups on a fingerprint of those files: a kubelet restart with unchanged config emits
# nothing, while a CSE retry that rewrote them re-emits so the event never reports a stale config.

[Service]
Type=oneshot
Restart=no
RemainAfterExit=no
ExecStart=/bin/bash /opt/azure/containers/emit-kubelet-active-flags.sh

vhdbuilder/packer/pre-install-dependencies.sh:85

• systemctl enable emit-kubelet-active-flags.service is not checked for failure. If enabling fails during the VHD bake, the telemetry will silently not run on nodes. This script generally treats systemd operations as critical (many earlier calls || exit 1), so it would be safer to fail fast here as well.

# Pulled in by kubelet.service via WantedBy=kubelet.service, so CSE does not need to start it.
systemctl enable emit-kubelet-active-flags.service

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (2)

e2e/validators.go:962

• The guard uses execOnVMForScenarioOnUnprivilegedPod(), which executes inside the debugnonhost daemonset pod. That pod is not host-mounted and typically does not have access to systemd, so systemctl cat ... will fail even when the unit exists and the validation will always be skipped. Use the SSH-based execScriptOnVMForScenario() for this host-level check.

	// Guard: skip on VHDs that don't have the service
	check := execOnVMForScenarioOnUnprivilegedPod(ctx, s,
		"systemctl cat emit-kubelet-active-flags.service 2>/dev/null")
	if check.exitCode != "0" {
		s.T.Log("emit-kubelet-active-flags.service not on this VHD, skipping validation")
		return
	}

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:75

• The PR description and Kusto example use the schema keys kubelet_flags_count / kubelet_flags, but the emitted payload uses flag_count / flags. This will break copy/paste queries and any downstream tooling expecting the documented schema; either align the payload field names to the documented schema or update the PR docs/tests accordingly.

    payload="$(jq -cn \
        --argjson uses_config_file "${uses_config_file}" \
        --arg config_path "${config_path}" \
        --argjson flag_count "${flag_count}" \
        --argjson flags "${flags_json}" \
        --argjson config_file "${config_file_json}" \
        '{uses_config_file: $uses_config_file, config_path: $config_path, flag_count: $flag_count, flags: $flags, config_file: $config_file}')"

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (3)

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.service:14

• RemainAfterExit=yes keeps a oneshot unit in the "active" state after it runs, so subsequent kubelet restarts in the same boot will not re-run this unit. That prevents the telemetry event from being refreshed when kubelet is restarted with new flags/config (e.g., after a CSE retry), leaving the emitted config potentially stale.

Type=oneshot
Restart=no
RemainAfterExit=yes
ExecStart=/bin/bash /opt/azure/containers/emit-kubelet-active-flags.sh

e2e/validators.go:971

• The guard uses execOnVMForScenarioOnUnprivilegedPod to run systemctl cat ..., but that executes inside a regular debug pod (bash -c) rather than on the node/VM, so it will likely fail (no systemd/systemctl access) and skip validation even when the unit exists. Also, the event-file check uses a pipeline without pipefail and xargs cat, which can succeed even when no event file is found (and journalctl likely needs sudo).

func ValidateKubeletActiveFlagsEvent(ctx context.Context, s *Scenario) {
	s.T.Helper()
	// Guard: skip on VHDs that don't have the service
	check := execOnVMForScenarioOnUnprivilegedPod(ctx, s,
		"systemctl cat emit-kubelet-active-flags.service 2>/dev/null")
	if check.exitCode != "0" {
		s.T.Log("emit-kubelet-active-flags.service not on this VHD, skipping validation")
		return
	}
	command := []string{
		"set -ex",
		// Verify service completed successfully via journalctl
		`journalctl -u emit-kubelet-active-flags.service --no-pager | grep -q "Finished\|Deactivated successfully"`,
		// Verify the event file was produced with correct TaskName
		`grep -rl 'kubeletActiveFlags' /var/log/azure/Microsoft.Azure.Extensions.CustomScript/events/ | head -1 | xargs cat | jq -e '.TaskName == "AKS.CSE.ensureKubelet.kubeletActiveFlags"'`,
	}
	execScriptOnVMForScenarioValidateExitCode(ctx, s, strings.Join(command, "\n"), 0, "failed to validate emit-kubelet-active-flags.service")
}

parts/linux/cloud-init/artifacts/emit-kubelet-active-flags.sh:76

• The emitted JSON schema here uses flag_count and flags, but the PR description and Kusto example query use kubelet_flags_count and kubelet_flags. This is a breaking mismatch for anyone using the documented query/payload shape; either the code or the PR description/query needs to be updated so they match.

    payload="$(jq -cn \
        --argjson uses_config_file "${uses_config_file}" \
        --arg config_path "${config_path}" \
        --argjson flag_count "${flag_count}" \
        --argjson flags "${flags_json}" \
        --argjson config_file "${config_file_json}" \
        '{uses_config_file: $uses_config_file, config_path: $config_path, flag_count: $flag_count, flags: $flags, config_file: $config_file}')"

[abigailliang-aks-sig-node]

aks-log-collector includes /etc/default/kubelet as a raw file in a ZIP bundle uploaded on-demand for support cases. This PR emits the same data as a structured JSON event to GuestAgentGenericLogs, enabling fleet-wide Kusto queries (e.g. "which nodes have maxPods != 250"). Both sides are reading the same document.