Skip to content

[Bug]: Beats setup tasks (filebeat/metricbeat) inconsistent with auditbeat #508

Description

@afeefghannam89

The setup tasks that push cluster-wide assets (index template, ILM, ingest pipelines, dashboards) into Elasticsearch differ across the three Beats, and each has gaps. Auditbeat is closest to correct.

Current state

Beat ES-output gated? run_once? flags problem
auditbeat yes yes --pipelines --index-management reference; only lacks re-run on change
metricbeat yes no bare setup re-pushes from every host; bare setup also pulls in --dashboards (needs Kibana); pointless with_items loop
filebeat no no --pipelines fails on the default logstash output; no --index-management; modules live in modules.d/ so setup --pipelines finds none (needs --modules); per-module marker

Background (how setup behaves)

Any explicit flag makes setup run ONLY those flags (bare setup = index-management + dashboards + ML, no pipelines). With the Elasticsearch output the index template and pipelines auto-load on first run anyway; with the Logstash output nothing auto-loads, so setup is really only needed there — yet the tasks are gated on the ES output and skip exactly that case. --index-management is safe (setup.template.overwrite: false, so it never clobbers a user template). --dashboards is the only Kibana-dependent part (and setup.kibana is currently empty — separate issue).

Target — make all three consistent

  • Gate on beats_<beat>_output == "elasticsearch".
  • Use explicit --index-management --pipelines; make --dashboards an opt-in toggle (only when Kibana is configured).
  • filebeat: pass --modules {{ ... }} (or --enable-all-filesets) so pipelines load for the modules enabled in modules.d/.
  • Replace the run-once-forever marker with re-run on change: register the enable/template task and run setup when: <register> is changed (+ run_once), so newly added modules/filesets get their pipelines. Drop metricbeat's pointless with_items.
  • Drop "(experimental)" from the beats_filebeat_modules description.

Note: the module handling is redesigned data-driven in #510 / #505 / #516 (inline, one list = desired state) — that further simplifies this (inline → setup --pipelines sees the modules directly, no --modules). This issue is the fix for the current architecture.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions