The setup tasks that push cluster-wide assets (index template, ILM, ingest pipelines, dashboards) into Elasticsearch differ across the three Beats, and each has gaps. Auditbeat is closest to correct.
Current state
| Beat |
ES-output gated? |
run_once? |
flags |
problem |
| auditbeat |
yes |
yes |
--pipelines --index-management |
reference; only lacks re-run on change |
| metricbeat |
yes |
no |
bare setup |
re-pushes from every host; bare setup also pulls in --dashboards (needs Kibana); pointless with_items loop |
| filebeat |
no |
no |
--pipelines |
fails on the default logstash output; no --index-management; modules live in modules.d/ so setup --pipelines finds none (needs --modules); per-module marker |
Background (how setup behaves)
Any explicit flag makes setup run ONLY those flags (bare setup = index-management + dashboards + ML, no pipelines). With the Elasticsearch output the index template and pipelines auto-load on first run anyway; with the Logstash output nothing auto-loads, so setup is really only needed there — yet the tasks are gated on the ES output and skip exactly that case. --index-management is safe (setup.template.overwrite: false, so it never clobbers a user template). --dashboards is the only Kibana-dependent part (and setup.kibana is currently empty — separate issue).
Target — make all three consistent
- Gate on
beats_<beat>_output == "elasticsearch".
- Use explicit
--index-management --pipelines; make --dashboards an opt-in toggle (only when Kibana is configured).
- filebeat: pass
--modules {{ ... }} (or --enable-all-filesets) so pipelines load for the modules enabled in modules.d/.
- Replace the run-once-forever marker with re-run on change:
register the enable/template task and run setup when: <register> is changed (+ run_once), so newly added modules/filesets get their pipelines. Drop metricbeat's pointless with_items.
- Drop "(experimental)" from the
beats_filebeat_modules description.
Note: the module handling is redesigned data-driven in #510 / #505 / #516 (inline, one list = desired state) — that further simplifies this (inline → setup --pipelines sees the modules directly, no --modules). This issue is the fix for the current architecture.
The
setuptasks that push cluster-wide assets (index template, ILM, ingest pipelines, dashboards) into Elasticsearch differ across the three Beats, and each has gaps. Auditbeat is closest to correct.Current state
run_once?--pipelines --index-managementsetupsetupalso pulls in--dashboards(needs Kibana); pointlesswith_itemsloop--pipelines--index-management; modules live inmodules.d/sosetup --pipelinesfinds none (needs--modules); per-module markerBackground (how
setupbehaves)Any explicit flag makes
setuprun ONLY those flags (baresetup= index-management + dashboards + ML, no pipelines). With the Elasticsearch output the index template and pipelines auto-load on first run anyway; with the Logstash output nothing auto-loads, sosetupis really only needed there — yet the tasks are gated on the ES output and skip exactly that case.--index-managementis safe (setup.template.overwrite: false, so it never clobbers a user template).--dashboardsis the only Kibana-dependent part (andsetup.kibanais currently empty — separate issue).Target — make all three consistent
beats_<beat>_output == "elasticsearch".--index-management --pipelines; make--dashboardsan opt-in toggle (only when Kibana is configured).--modules {{ ... }}(or--enable-all-filesets) so pipelines load for the modules enabled inmodules.d/.registerthe enable/template task and runsetupwhen: <register> is changed(+run_once), so newly added modules/filesets get their pipelines. Drop metricbeat's pointlesswith_items.beats_filebeat_modulesdescription.Note: the module handling is redesigned data-driven in #510 / #505 / #516 (inline, one list = desired state) — that further simplifies this (inline →
setup --pipelinessees the modules directly, no--modules). This issue is the fix for the current architecture.