Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions content/en/logs/explorer/advanced_search.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,16 @@ further_reading:
- link: "/logs/explorer/saved_views/"
tag: "Documentation"
text: "Learn about Saved Views"
- link: "/monitors/types/log/"
tag: "Documentation"
text: "Create Log Monitors"
---

## Overview

When you need to further refine the results of a log search, use [subqueries](#filter-logs-with-subqueries) to compare your results against a secondary group of logs, or filter your logs using data from your [Reference Tables](#filter-logs-based-on-reference-tables).
When you need to further refine log search results or build curated datasets for analysis and monitoring, use [subqueries](#filter-logs-with-subqueries) to compare your results against a secondary group of logs, or filter your logs using data from your [Reference Tables](#filter-logs-based-on-reference-tables).

Advanced search options are available in the Log Explorer and [log monitors][3].

## Filter logs with subqueries

Expand All @@ -41,7 +46,7 @@ This introduces new elements to the query editor:
1. Select whether you want to use the {{< ui >}}IN{{< /ui >}} or {{< ui >}}NOT IN{{< /ui >}} operator:
- The results from the {{< ui >}}IN{{< /ui >}} operator only include logs where the attribute value is also found in the subquery results. For example, if you only want to see logs generated by `service:a` and is associated to a user who is also one of the top users of `service:b`.
- The results from the {{< ui >}}NOT IN{{< /ui >}} operator exclude logs where the attribute value is found in the subquery results. For example, if you only want to see `status:error` logs, but also want to filter out `status:error` logs where users associated with those logs eventually end up with a `status:success` log. See [Filter outdated or superseded logs](#filter-outdated-or-superseded-logs) for a detailed example.
1. Optionally, reduce the number of subquery attribute values to match against. The default value and maximum value is `1000`. Choose between {{< ui >}}top{{< /ui >}} (highest frequency values) or {{< ui >}}bottom{{< /ui >}} (lowest frequency values).
1. Optionally, reduce the number of subquery attribute values to match against. The default value is `1000` and the maximum value is `1,000,000`. Choose between {{< ui >}}top{{< /ui >}} (highest frequency values) or {{< ui >}}bottom{{< /ui >}} (lowest frequency values).

### Subquery examples

Expand Down Expand Up @@ -84,3 +89,4 @@ In this example, use `service:network_directory` as your main query, and then de

[1]: https://app.datadoghq.com/logs
[2]: /integrations/guide/reference-tables/?tab=manualupload
[3]: /monitors/types/log/
18 changes: 14 additions & 4 deletions content/en/monitors/types/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ further_reading:

Logs are essential for security investigations, aiding in threat detection, compliance tracking, and security monitoring. Log Management systems correlate logs with observability data for rapid root cause detection. Log management also enables efficient troubleshooting, issue resolution, and security audits.

Once [log management is enabled][1] for your organization, you can create a logs monitor to alert you when a specified log type exceeds a user-defined threshold over a given period of time. The logs monitor only evaluates [indexed logs][2].
Once [log management is enabled][1] for your organization, you can create a logs monitor to alert you when a specified log type exceeds a user-defined threshold over a given period of time. The logs monitor only evaluates [indexed logs][2].

**Note**: Log monitors have a maximum rolling time window of `2 days`.

Expand Down Expand Up @@ -67,7 +67,13 @@ As you define the search query, the graph above the search fields updates.
* **Third facet**: 5 top values
* **Fourth facet**: 2 top values

4. Configure the alerting grouping strategy (optional):
4. Optionally, use the {{< ui >}}Σ Modify Query{{< /ui >}} menu to refine the query with:
* **Functions and formulas**: Transform and combine your query results. See [functions][11] and [formulas][12].
* **Advanced search**: Scope alerts to curated datasets, such as entity lists, customer tiers, or correlated events. See [subqueries][13] and [Reference Tables][14].

You can also create a monitor directly from an advanced search in the [Log Explorer][4].

5. Configure the alerting grouping strategy (optional):
* {{< ui >}}Simple-Alert{{< /ui >}}: Simple alerts aggregate over all reporting sources. You receive one alert when the aggregated value meets the set conditions. This works best to monitor a metric from a single host or the sum of a metric across many hosts. This strategy may be selected to reduce notification noise.
* {{< ui >}}Multi Alert{{< /ui >}}: Multi alerts apply the alert to each source according to your group parameters. An alerting event is generated for each group that meets the set conditions. For example, you could group `system.disk.in_use` by `device` to receive a separate alert for each device that is running out of space.

Expand All @@ -83,9 +89,9 @@ Trigger when the query meets one of the following conditions compared to a thres

#### No data and below alerts

`NO DATA` is a state given when no logs match the monitor query during the timeframe.
`NO DATA` is a state given when no logs match the monitor query during the time frame.

To receive a notification when all groups matching a specific query have stopped sending logs, set the condition to `below 1`. This notifies when no logs match the monitor query in a given timeframe across all aggregate groups.
To receive a notification when all groups matching a specific query have stopped sending logs, set the condition to `below 1`. This notifies when no logs match the monitor query in a given time frame across all aggregate groups.

When splitting the monitor by any dimension (tag or facet) and using a `below` condition, the alert is triggered **if and only if** there are logs for a given group, and the count is below the threshold—or if there are no logs for **all** of the groups.

Expand Down Expand Up @@ -143,3 +149,7 @@ Include a sample of 10 logs in the alert notification:
[8]: /monitors/notify/
[9]: /monitors/notify/variables/?tab=is_alert#matching-attributetag-variables
[10]: /logs/log_configuration/flex_logs/
[11]: /logs/explorer/analytics/#functions
[12]: /logs/explorer/analytics/#formulas
[13]: /logs/explorer/advanced_search/#filter-logs-with-subqueries
[14]: /logs/explorer/advanced_search/#filter-logs-based-on-reference-tables
Loading