Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .dev-loop/INGEST_REPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Knowledge flush — 2 insight(s) (+1 retired as already flushed)

Cross-Check: 독립 서브에이전트 적대검증 1회 — date 페이지 CONFIRMED(GNU coreutils 컨테이너·macOS 26.5.1 양쪽 실측, case 패턴 분기 검증), entrypoint 페이지는 큐 원문의 TERM trap 근거가 실증상 반대(무trap SIGTERM에도 bash EXIT trap 실행; foreground 자식 뒤 TERM trap은 지연→SIGKILL 137, flush 0건)임을 적발 → background 자식+`wait`+시그널 포워딩 패턴으로 재작성 후 반영. 재현 아티팩트: bash 4.4/5.2 PID-1 docker 실험 각 10회.

## Decision Log

- **의도**: dev-loop 큐의 pending 인사이트를 검증·중복점검·라우팅해 위키에 반영 (knowledge-flush 파이프라인, 사용자가 flush 실행과 PR 1건 생성을 명시 지시).
- **배제한 대안**:
- 인사이트(kubelet /metrics/resource)의 신규 페이지 생성 — **배제: 열려 있는 PR #2가 동일 인사이트를 `container-metrics-when-cadvisor-is-empty.md`로 이미 반영**(metrics_path 핀 + image!="" 복합 필터까지 커버 확인). 큐 row가 retire되지 않아 재등장한 것 — 이번 flush에서 재반영 없이 retire만 함. 작성했던 중복 페이지는 커밋 이력에서 제거.
- 인사이트(PID-1 tee)를 `platforms/shells/portable-shell-scripts.md`에 병합 — 배제: 트리거가 셸 이식성이 아니라 PID-1 컨테이너 시맨틱이라 `infrastructure/containers` 신규 페이지가 맞음 (one case per page).
- 큐 원문 directive(`trap 'exit 143' TERM` 단독) 그대로 반영 — 배제: 적대검증 실험에서 반박되어 교정 후 반영.
- **리뷰어가 볼 곳**: `entrypoint-log-capture.md` step 2(background-child+wait 패턴 — 교차검증으로 재작성된 부분)와 Sources의 두 독립 재현 기록, `bsd-vs-gnu-cli.md` 추가 행의 case 패턴 분기. `wiki/infrastructure/index.md`는 PR #2도 수정하므로 머지 순서에 따라 사소한 충돌 가능.
- [추정] RTB 팀 표준 리뷰어 3인은 이 개인 레포 collaborator가 아니어서 리뷰어 지정이 거부됨(gh 확인: `d43103-rsquare not found`) — 레포 소유자 리뷰(스킬 기본 모델)로 진행.

## Verified best-practice

**1. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"`; long foreground jobs need background-child + `wait` + TERM forwarding.**
Verified against:
- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; managed with `wait "$!"` since bash 4.4 (also confirmed in bash CHANGES, bash-4.4 section).
- https://tiswww.case.edu/php/chet/bash/bashref.html — 128+n exit status for signal-terminated commands (143 = SIGTERM); trap execution deferred until the running foreground command completes.
- Session reproduction (OrbStack container): without trap 0/10 runs captured output; with trap 10/10.
- Independent adversarial re-reproduction during this flush (docker, bash 4.4 and 5.2 as PID 1): 1/10 without trap vs 10/10 with; untrapped `docker stop` ran the EXIT trap (exit 143); `trap 'exit 143' TERM` behind a foreground child was deferred until SIGKILL (exit 137, no flush).
The queued candidate's original `trap 'exit 143' TERM` directive was **corrected during cross-check** — the page teaches the working background-child pattern instead. **confidence: verified.** (gnu.org bash manual rate-limited with HTTP 429 during this flush; the case.edu mirror + Wooledge are cited instead — no unverified URL included.)

**2. BSD/GNU `date` ms-timestamp feature detection → test the actual `%3N` output shape, not `%N` presence.**
Verified against:
- https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — "%N nanoseconds … This is a GNU extension."
- https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension.
- Independent reproduction on a second machine during this flush (macOS 26.5.1, beyond the original 14.8.3 measurement): `date +%N` → digits, `date +%3N` → literal `3N`. Adversarial re-check additionally ran GNU coreutils (debian container): `date +%3N` → exactly 3 digits, and confirmed the `case … [0-9][0-9][0-9])` pattern takes the GNU path on GNU and falls through on macOS.
**confidence: verified.**

**Retired without re-ingesting**: the OrbStack kubelet `/metrics/resource` insight (hash `ab44b921a1f3f759`) — already flushed as open **PR #2** (`container-metrics-when-cadvisor-is-empty.md`) but never removed from its session queue file; verified PR #2's page already covers the chart defaults, the label-set difference, and the `metrics_path="/metrics/cadvisor"` + `image!=""` dashboard exclusion, so nothing new to add.

## Existing-layer check

Read: root `INDEX.md`, `wiki/infrastructure/index.md`, `wiki/platforms/index.md`, and every overlapping page: `observability/logs-metrics-signals.md`, `containers/resource-limits-and-probes.md`, `containers/image-builds.md`, `shells/portable-shell-scripts.md`, `tools/bsd-vs-gnu-cli.md`, plus open PR #2's `container-metrics-when-cadvisor-is-empty.md` (via the PR branch).

- Insight 1 (PID-1 tee): no overlap — `portable-shell-scripts` covers shell portability, `image-builds` covers Dockerfiles; neither covers PID-1 runtime log/signal behavior. No conflict. **Created new page**; related-linked both ways to `image-builds` and `portable-shell-scripts` (cross-domain), one-way to `logs-metrics-signals`.
- Insight 2 (date `%3N`): direct overlap with `platforms/tools/bsd-vs-gnu-cli.md` (same trigger family — its "Relative date" row already covers `date -d` vs `-v`). **Merged**: one command-table row (sub-second timestamp), one edge-case row (`%N` printing digits on macOS must not imply GNU), two sources, `last_verified` bumped to 2026-07-15. No new page.
- Kubelet insight: duplicate of open PR #2 → dropped (see Decision Log).

Conflicts flagged: none in content. Noted: `wiki/infrastructure/index.md` is also touched by open PR #2 (different rows) — trivial merge-order conflict possible.

## Routing decision

| Insight | Target | New category? |
|---------|--------|---------------|
| PID-1 tee log loss | `infrastructure/containers/entrypoint-log-capture.md` (new page) | No — containers owns container-runtime behavior; platforms/shells rejected because the trigger is PID-1 container semantics, not shell portability |
| date `%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (merge) | No — exact existing page for BSD-vs-GNU flag differences |
| kubelet `/metrics/resource` | — (retired; already in open PR #2) | — |

Nothing left `unverified`. Queue rows retired to `~/.dev-loop/queue/.processed.jsonl` after PR creation.
2 changes: 2 additions & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,5 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-07-11] ingest | security +1 (new category api-exposure): exposing-an-origin-http-api — code-vs-edge control split for putting an origin HTTP API behind a public tunnel/proxy. Captures the non-obvious 5xx-bypasses-request-middleware gotcha (set security headers in the exception handler too — verify with a forced 500), header-only fail-closed auth, docs/OpenAPI off at the origin by default, sanitized 5xx bodies, and the code(validate/headers/errors)-vs-edge(TLS/HSTS/rate-limit/WAF) boundary. Derived from hardening korea-data-suite for external exposure + independent pentest; sources OWASP Secure Headers, MDN, Starlette middleware, FastAPI docs.
## [2026-07-12] ingest | databases +1 (new category sqlite): concurrent-access-for-a-read-api — WAL + busy_timeout for a read-heavy API with a background writer; set journal_mode once (per-connection pragma cost ~0.4ms measured vs 0.04ms busy_timeout), single writer, run the batch writer as a separate process (interpreter-lock isolation), scale reads with worker processes not threads. Derived from load-testing korea-data-suite (measured connect/query/pragma costs, GIL-bound concurrency, separate-sync-process deployment). Sources: SQLite WAL/pragma/FAQ docs.
## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
## [2026-07-15] ingest | infrastructure-containers-entrypoint-log-capture — PID-1 entrypoint bash with exec > >(tee …) loses log output because the shell never waits for process substitutions and the container dies with PID 1; fix is EXIT-trap fd close + wait "$TEE_PID"; for long foreground jobs, background-child + wait + TERM forwarding (a bare TERM trap behind a foreground child is deferred until SIGKILL — adversarial cross-check corrected the queued directive). Sources: Wooledge ProcessSubstitution (wait "$!" since bash 4.4), bash ref manual 128+n; field reproduction 0/10 → 10/10
## [2026-07-15] revise | platforms-tools-bsd-vs-gnu-cli +1 row +1 edge case: sub-second timestamps — recent macOS date prints digits for %N but emits %3N literally as "3N", so feature-detect on the actual %3N output shape, never on %N presence. GNU manual confirms %N and field width are GNU extensions; reproduced on macOS 14.8.3 and 26.5.1. last_verified bumped to 2026-07-15
83 changes: 83 additions & 0 deletions wiki/infrastructure/containers/entrypoint-log-capture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
id: infrastructure-containers-entrypoint-log-capture
domain: infrastructure
category: containers
applies_to: [docker, kubernetes, bash]
confidence: verified
sources:
- https://mywiki.wooledge.org/ProcessSubstitution
- https://tiswww.case.edu/php/chet/bash/bashref.html
last_verified: 2026-07-15
related: [infrastructure-containers-image-builds, infrastructure-observability-logs-metrics-signals, platforms-shells-portable-shell-scripts]
---

# Log Loss from Process-Substituted tee in Container Entrypoint Scripts

## When this applies

A container's entrypoint bash script (running as PID 1) duplicates its output to
a file with `exec > >(tee -a "$LOG") 2>&1`, and log lines are missing: short jobs
capture nothing, long jobs lose the tail (the completion-summary lines).

## Do this

1. Capture the tee PID and wait for it in an EXIT trap. Bash runs process
substitutions asynchronously and does not wait for them; `wait "$!"` on a
process substitution is supported since bash 4.4. Outside a container the
orphaned tee usually finishes anyway — as PID 1, the container is torn down
the instant the script exits, killing tee before it flushes:

```bash
exec > >(tee -a "$LOG") 2>&1
TEE_PID=$!
flush_logs() {
exec 1>&- 2>&- # close our ends so tee sees EOF
wait "$TEE_PID"
}
trap flush_logs EXIT
```

2. For a script whose main work is a long foreground job, run that job as a
background child and `wait` it before adding any TERM trap. Bash defers
user traps until the foreground command completes — a TERM trap behind a
foreground child never fires on `docker stop`, the grace period expires,
and SIGKILL ends the container at exit 137 with nothing flushed. `wait`
is interruptible by traps, so this shape reacts immediately:

```bash
trap 'kill -TERM "$MAIN_PID" 2>/dev/null; wait "$MAIN_PID"; exit 143' TERM
main_job & MAIN_PID=$!
wait "$MAIN_PID"
```

`exit 143` (128+15) reports the conventional SIGTERM status. For simple
sequential scripts, no TERM trap is needed for the flush itself — bash runs
the EXIT trap on an untrapped SIGTERM in non-interactive shells.

3. Verify by re-running the container several times and checking the log file
for the final line each run — this failure is timing-dependent, so a single
successful run proves nothing.

## Edge cases

| Case | Then |
|------|------|
| Base image bash is older than 4.4 (`wait "$!"` on a process substitution fails) | Use a named pipe: `mkfifo`, start `tee < pipe &` as a normal background job, `exec > pipe`, and `wait` its real PID |
| Script also traps other signals (INT, HUP) | End each handler with `exit 128+n`, so every trapped path reports its conventional signal code and reaches the EXIT trap |
| Logs must survive even a SIGKILL (OOMKill, forced deletion) | No trap runs on KILL — write to a mounted volume via unbuffered append (or ship lines as they are produced) instead of relying on exit-time flushing |

## Instead of

| If you are about to | Do this instead | Why |
|---------------------|-----------------|-----|
| Trust `exec > >(tee -a f)` because it works in a terminal | Add the EXIT-trap wait before it runs as PID 1 | The shell never waits for process substitutions; a terminal leaves the orphan running to finish, a container kills it at PID 1 exit |
| Debug the missing lines as a tee buffering option problem | Reproduce with repeated short runs and add the trap | The writer is killed before flushing — buffering flags don't fix a killed process |
| Exit 0 from the TERM trap for a "clean" shutdown | `exit 143` (128+15) | Orchestrators and CI distinguish signalled termination from success by the 128+n convention |
| Add `trap 'exit 143' TERM` while the main job runs in the foreground | Background the job, `wait` it, and forward TERM from the trap (step 2) | Bash defers traps until the foreground command finishes — the trap never fires, the grace period ends in SIGKILL (exit 137), and nothing flushes |

## Sources

- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"`
- https://tiswww.case.edu/php/chet/bash/bashref.html — exit status is 128+n for a command terminated by signal n; trap execution is deferred until the running foreground command completes
- Field reproduction (OrbStack container, 2026-07-15): without the trap the log captured the script output in 0 of 10 runs; with the EXIT-trap wait, 10 of 10
- Independent adversarial re-reproduction (docker bash 4.4 and 5.2 as PID 1, 2026-07-15): tail captured 1/10 without the trap vs 10/10 with it; untrapped `docker stop` ran the EXIT trap (exit 143); `trap 'exit 143' TERM` behind a foreground child was deferred until SIGKILL (exit 137, no flush)
2 changes: 1 addition & 1 deletion wiki/infrastructure/containers/image-builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ sources:
- https://docs.docker.com/build/building/best-practices/
- https://docs.docker.com/build/building/secrets/
last_verified: 2026-07-10
related: [infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secrets-handling]
related: [infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secrets-handling, infrastructure-containers-entrypoint-log-capture]
---

# Structuring Container Image Builds
Expand Down
1 change: 1 addition & 0 deletions wiki/infrastructure/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Match your situation to a "load when" line; load only matching pages.
|------|-----------|
| [image-builds](containers/image-builds.md) | Writing or reviewing a Dockerfile; images rebuild everything on small changes, build slowly, or are too large; choosing an image tagging scheme |
| [resource-limits-and-probes](containers/resource-limits-and-probes.md) | Writing or reviewing Kubernetes-style deployment manifests; pods OOMKilled, evicted, or CPU-throttled; a dependency outage triggered a restart storm; traffic hitting pods that are not ready |
| [entrypoint-log-capture](containers/entrypoint-log-capture.md) | A container entrypoint bash script tees its output to a log file and lines are missing (short jobs capture nothing, long jobs lose the tail); wiring stdout duplication or signal traps into a PID-1 shell script |

## data

Expand Down
2 changes: 1 addition & 1 deletion wiki/platforms/shells/portable-shell-scripts.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ sources:
- https://google.github.io/styleguide/shellguide.html
- https://www.shellcheck.net/
last_verified: 2026-07-10
related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management]
related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, infrastructure-containers-entrypoint-log-capture]
---

# Shell Scripts That Must Run on More Than One Machine or Shell
Expand Down
7 changes: 6 additions & 1 deletion wiki/platforms/tools/bsd-vs-gnu-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sources:
- https://man.freebsd.org/cgi/man.cgi?date(1)
- https://man.freebsd.org/cgi/man.cgi?sed(1)
- https://man.freebsd.org/cgi/man.cgi?seq(1)
last_verified: 2026-07-10
last_verified: 2026-07-15
related: [platforms-shells-portable-shell-scripts]
---

Expand All @@ -32,6 +32,7 @@ flags do not. Apply the portable fix per command:
| `timeout` | In coreutils | Absent on stock macOS — exits 127 | Install coreutils and call `gtimeout`, or background the command and `kill` it from a sleep watchdog |
| `seq -s,` | Separator between numbers only | Separator also emitted after the last number, before the terminator | Join after generating (`seq 1 3 \| paste -sd, -`) and count items by lines, never by separators |
| PCRE grep | `grep -P` | Not supported | `grep -E` with ERE, or `perl -ne 'print if /…/'` |
| Sub-second timestamp | `date +%3N` → milliseconds (`%N` and the field width are both GNU extensions) | Recent macOS prints digits for `%N` but emits `%3N` literally as `3N` (timestamps become `…08.3NZ`) | Feature-detect on the actual `%3N` output, not on `%N`: `case "$(date +%3N)" in [0-9][0-9][0-9]) …GNU path… ;; *) …fallback… ;; esac`; or `gdate +%3N`, or settle for epoch seconds |
| Resolve path | `readlink -f` | Absent before macOS 13 | `cd "$(dirname "$f")" && pwd -P` for directories, `python3 -c 'import os,sys;print(os.path.realpath(sys.argv[1]))'`, or coreutils `greadlink -f` |
| File metadata | `stat -c '%s'` | `stat -f '%z'` | Detect once: `stat -c %s "$f" 2>/dev/null \|\| stat -f %z "$f"` |

Expand All @@ -49,6 +50,7 @@ General strategy by situation:
| Case | Then |
|------|------|
| `command -v timeout` succeeds on macOS | Someone installed coreutils unprefixed — confirm `timeout --version` reports GNU coreutils before relying on GNU exit-code semantics (124 on timeout) |
| `date +%N` prints digits on macOS | Do not infer GNU userland from one specifier working — width-modified forms (`%3N`) still fail on BSD `date`; feature-detect the exact format string you will use, by its output shape |
| Any flags passed to `echo` (`-e`, `-n`) | `echo` flag handling differs across shells and userlands — use `printf` for anything beyond a bare literal string |
| Script needs bash 4+ features on macOS | Stock `/bin/bash` on macOS is 3.2 — use `#!/usr/bin/env bash` so a brew-installed bash is picked up, and state the required bash version in the script header |

Expand All @@ -63,6 +65,9 @@ General strategy by situation:
## Sources

- https://www.gnu.org/software/coreutils/manual/html_node/index.html — GNU date/timeout/seq/stat behavior
- https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — `%N` (nanoseconds) is a GNU extension
- https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension
- Field reproduction (macOS 14.8.3 and 26.5.1, 2026-07-15): `date +%N` → digits, `date +%3N` → literal `3N` on both
- https://man.freebsd.org/cgi/man.cgi?date(1) — BSD `date -v` adjustment flag
- https://man.freebsd.org/cgi/man.cgi?sed(1) — BSD `sed -i` backup-extension argument
- https://man.freebsd.org/cgi/man.cgi?seq(1) — BSD `seq -s` separator emitted after the last number (see man-page example)