Observability: surface the real failure cause instead of generic "local-exec provisioner error" / "paused due to internal error"

[vsilent]

Problem

The diagnostic information needed to understand a failed deploy exists at every layer of the pipeline (Docker daemon, Ansible task stderr, Terraform stderr, cloud-provider API responses) — but stacker collapses it into a generic wrapper string before it reaches the operator. The result: when a deploy pauses, stacker status / stacker deployment events tell you that it failed but not why, and you're left retrying blindly.

This generalizes #222 (which fixed one specific Hetzner case at the --watch layer) into the broader error-surfacing story.

Concrete motivating example (this week)

A cloud deploy paused. The complete output of stacker deployment events was:

1. [deployment_status / paused] Deployment has been paused due to internal error.
   Details: Application Stack: healthchecks
   Error: local-exec provisioner error

The actual cause was only visible in the raw Terraform/Ansible subprocess log (watchers.py:858 execute_tf()), which the CLI does not expose:

Error response from daemon: driver failed programming external connectivity on endpoint
project-app-1 (...): Bind for 0.0.0.0:5000 failed: port is already allocated
...
PLAY RECAP: 88.198.151.116 : ok=152 changed=100 unreachable=0 failed=1

i.e. a plain host port conflict on 5000 (the statuspanel was already bound there) — a specific, actionable, single-line cause, thrown away and replaced with local-exec provisioner error.

This is a recurring pattern, not a one-off. Other instances seen the same way:

What stacker shows	Real cause (found only outside stacker)
local-exec provisioner error	Bind for 0.0.0.0:5000 failed: port is already allocated
paused due to internal error (Ansible log truncated after "Show me OS")	container host-port already bound by another stack
Application Stack: <name>	the app container's own crash log (e.g. role "x" does not exist, missing SETTINGS_ENCRYPTION_KEY)

Proposed improvements (roughly priority order)

1. Surface the real stderr. stacker deployment events / stacker status should include the last failing task's actual stdout/stderr (the Ansible msg/stdout_lines, Terraform stderr) — even just the last ~20 lines — instead of a hand-written summary. This alone turns "retry and hope" into diagnosis.

2. Remote preflight, mirroring the local one. --target local already runs a preflight that catches host-port conflicts cleanly before deploying (e.g. port 8000 (service 'app') is already allocated on this host — find the owner with: lsof ...). That same check does not run for --target server/--target cloud, so a port-5000 conflict only surfaces after provisioning an entire server that then pauses. Port the preflight (ports free? disk space? images pullable?) to the remote host.

3. Classify known, recurring failure modes into typed errors + remediation, e.g.:

• port is already allocated → E: Port <n> on <host> already in use (likely statuspanel/another stack). Free it or change the port.
• provider resource_limit_exceeded → quota message (partially done in Hetzner resource_limit_exceeded error not surfaced in --watch output #222 for one path — generalize it)
• Postgres role does not exist / database is uninitialized → point at the volume/secret cause
• SSH connect timeout after provisioning → E: server provisioned but unreachable on :22 (firewall not created / network issue)

4. Raw-log escape hatch. A stacker deployment events --raw (or stacker logs --deploy <hash>) that dumps the underlying install-service TF/Ansible log the backend already captured, for failed deploys — not just app logs for running ones.

5. Consistent terminal-state contract. Every paused/failed state should end with: severity + one-line real cause + a concrete next command. --target local already does this; the remote paths should match.

Impact

High for operability. Without this, diagnosing any remote deploy failure requires SSHing to the box and reading docker/journal logs by hand (often impossible if the failure is exactly that the box is unreachable), or having someone paste the backend's raw TF log. The information is already captured server-side — it's a plumbing/formatting gap, not a data-collection gap.

Environment

• stacker CLI 0.3.1 (144d1a6)
• Observed repeatedly across stacker-projects/* in trydirect/stacker-project-examples
• Related/prior art: Hetzner resource_limit_exceeded error not surfaced in --watch output #222 (closed, narrow Hetzner-quota variant of Create roadmap #1)

[vsilent]

Positive data point on the same session: for a cloud/Terraform failure, stacker status actually did surface a good chunk of the real Terraform apply trace, which is much closer to what this issue asks for:

Deployment has been paused. Error: Error: hcloud_network.network: Creating... |
hcloud_firewall.firewall: Creating... | hcloud_firewall.firewall: Creation complete after 1s [id=...] |
hcloud_server.server[0]: Creating... | hcloud_network.network: Creation complete ... |
hcloud_server.server[0]: Still creating... [10s elapsed] | hcloud_server.server[0

(plus An unclassified internal error occurred. Redeploying will retry; if it keeps failing, contact support with the deployment hash.)

So the Terraform-layer surfacing is already partly there and works well. The gap in this issue is specifically the Ansible/install-service (local-exec) layer — that's where it still collapses to the opaque local-exec provisioner error / Application Stack: <name> with the real task stderr (e.g. the port-5000 bind error) dropped. Worth mirroring the Terraform-trace treatment onto the Ansible-task failures too. Also note the TF trace here is truncated mid-line (hcloud_server.server[0 — cut off), so a bit more buffer / not truncating would help.