-
Notifications
You must be signed in to change notification settings - Fork 14
ful_SC-23656_cross_connect_synth_with_traces_logs_metrics #721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
fulyauluturk
wants to merge
12
commits into
master
Choose a base branch
from
ful_synth_cross_connect_and_new_app_flows
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
caaee8b
Update Synthetics getting started and SSL certificate monitoring pages
fulyauluturk 3998121
Rewrite Root Cause Discovery docs with Metrics, Logs and Traces corre…
fulyauluturk fbab9fd
Update adding-trace-id-to-response-headers.md
fulyauluturk 3734067
Update adding-trace-id-to-response-headers.md
fulyauluturk d063ef4
Clarify trace ID detection impact on log filtering
fulyauluturk 36c13e4
Update metrics-correlation.md
fulyauluturk 3f7bcfb
Update logs-correlation.md
fulyauluturk 857a4e9
Update traces-correlation.md
fulyauluturk c8e1fa1
Update adding-trace-id-to-response-headers.md
fulyauluturk 9673315
Update adding-trace-id-to-response-headers.md
fulyauluturk a1e1f05
Update logs-correlation.md
fulyauluturk 59d5582
Enhance tracing app setup instructions
fulyauluturk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+582 KB
docs/images/synthetics/root-cause-discovery/trace-id-response-header.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
109 changes: 109 additions & 0 deletions
109
docs/synthetics/root-cause-discovery/adding-trace-id-to-response-headers.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| title: Expose Trace ID in Response Headers | ||
| description: Expose your active trace ID in HTTP response headers so Sematext Synthetics can correlate failed monitor runs with the exact matching logs and traces. | ||
|
|
||
| When Sematext Synthetics runs a monitor and the request fails, it checks the response headers for a trace ID. If one is present, it uses it to filter logs and traces to that exact request. Without it, correlation falls back to URL and time window matching, which may include data from unrelated requests. | ||
|
|
||
| Adding a trace ID to your response headers is a small, one-time change to your application. The trace ID comes from your existing OpenTelemetry instrumentation — no additional setup is required. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Your service must already be instrumented with OpenTelemetry and shipping traces to a Sematext Tracing App. If you haven't set that up yet, see [Getting Started with Tracing](/docs/tracing/getting-started/) and the [OpenTelemetry SDKs](/docs/tracing/sdks/) documentation. | ||
|
|
||
| For complete working examples, see the [sematext-opentelemetry-examples](https://github.com/sematext/sematext-opentelemetry-examples) repository. | ||
|
|
||
| ## Java / Spring Boot | ||
|
|
||
| Add a servlet filter that reads the active span context from the OTel agent and writes the trace ID to the response headers before the request is processed. | ||
|
|
||
| ```java | ||
| import io.opentelemetry.api.trace.Span; | ||
| import io.opentelemetry.api.trace.SpanContext; | ||
| import jakarta.servlet.FilterChain; | ||
| import jakarta.servlet.ServletException; | ||
| import jakarta.servlet.http.HttpServletRequest; | ||
| import jakarta.servlet.http.HttpServletResponse; | ||
| import org.springframework.core.Ordered; | ||
| import org.springframework.core.annotation.Order; | ||
| import org.springframework.stereotype.Component; | ||
| import org.springframework.web.filter.OncePerRequestFilter; | ||
|
|
||
| import java.io.IOException; | ||
|
|
||
| @Component | ||
| @Order(Ordered.LOWEST_PRECEDENCE - 10) | ||
| public class TraceIdFilter extends OncePerRequestFilter { | ||
|
|
||
| @Override | ||
| protected boolean shouldNotFilter(HttpServletRequest request) { | ||
| return "/error".equals(request.getRequestURI()); | ||
| } | ||
|
|
||
| @Override | ||
| protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain) | ||
| throws ServletException, IOException { | ||
|
|
||
| SpanContext ctx = Span.current().getSpanContext(); | ||
| if (ctx.isValid()) { | ||
| response.setHeader("X-Trace-Id", ctx.getTraceId()); | ||
| } | ||
|
|
||
| chain.doFilter(request, response); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The OTel Java agent creates the span at the Tomcat connector level, above the filter chain, so `Span.current()` is already valid when the filter runs. Headers must be set before `chain.doFilter()` is called because the response is committed after that point. No additional dependency is needed beyond the OTel API, which is already on the classpath when using the Java agent. | ||
|
|
||
| ## Node.js / Express | ||
|
|
||
| Add a middleware that runs on every request and sets the trace ID header using the OTel API. | ||
|
|
||
| ```js | ||
| const { trace } = require('@opentelemetry/api'); | ||
|
|
||
| app.use((req, res, next) => { | ||
| const span = trace.getActiveSpan(); | ||
| if (span) { | ||
| res.setHeader('X-Trace-Id', span.spanContext().traceId); | ||
| } | ||
| next(); | ||
| }); | ||
| ``` | ||
|
|
||
| Register this middleware early in your Express app, before your route handlers. | ||
|
|
||
| ## Python / Flask | ||
|
|
||
| Use Flask's `after_request` hook to add the trace ID to every response. | ||
|
|
||
| ```python | ||
| from opentelemetry import trace | ||
|
|
||
| @app.after_request | ||
| def add_trace_id(response): | ||
| span = trace.get_current_span() | ||
| ctx = span.get_span_context() | ||
| if ctx.is_valid: | ||
| response.headers['X-Trace-Id'] = format(ctx.trace_id, '032x') | ||
| return response | ||
| ``` | ||
|
|
||
| ## Verifying It Works | ||
|
|
||
| How you verify the header depends on the Synthetics Monitor type. | ||
|
|
||
| **HTTP Monitor** — open the monitor's run details and check the **Request** tab. You should see the `X-Trace-Id` header in the response headers section. | ||
|
|
||
| **Browser Monitor** — open the monitor's run details and go to the **Waterfall** view. Click on the URL you instrumented, then open its **Request** tab to inspect the response headers for that specific request. | ||
|
|
||
|  | ||
|
|
||
| Once Sematext detects the trace ID, the Logs and Traces tabs in the Troubleshoot section will let you drill into the exact logs and the individual trace associated with the failed monitor run. Without trace ID, logs and traces will be filtered by the URL and time window, which could include logs and traces from other requests as well. | ||
|
|
||
| ## Further Reading | ||
|
|
||
| - [Logs Correlation](/docs/synthetics/root-cause-discovery/logs-correlation/) | ||
| - [Traces Correlation](/docs/synthetics/root-cause-discovery/traces-correlation/) | ||
| - [OpenTelemetry SDKs](/docs/tracing/sdks/) | ||
| - [Sematext OpenTelemetry Examples](https://github.com/sematext/sematext-opentelemetry-examples) | ||
| - [Getting Started with Tracing](/docs/tracing/getting-started/) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| title: Logs Correlation | ||
| description: Correlate failed Synthetics monitor runs with application and service logs. | ||
|
|
||
| The Logs tab in the failed run flyout finds logs from [connected](/docs/guide/connected-apps/) Logs Apps that match the failed request. Depending on your setup, logs are matched by exact trace ID or by URL and time window. | ||
|
|
||
| ## Connecting a Logs App | ||
|
|
||
| If you don't have a Logs App, the Logs section in the Troubleshoot tab will prompt you to create one and connect it to your Synthetics App automatically. If you already have Logs Apps in your account, you can select and connect the relevant one directly from the same tab. | ||
|
|
||
| We recommend using the [OpenTelemetry Logs integration](/docs/integration/opentelemetry-logs/) for two reasons: if you expose your trace ID in response headers, Sematext can filter logs to the exact failing request; and even without a trace ID, OTel logs are structured and enriched, making it easier to spot errors and correlate across services. The [Sematext OpenTelemetry Examples](https://github.com/sematext/sematext-opentelemetry-examples) repo includes log shipping examples alongside the tracing and metrics setup. | ||
|
|
||
| If Sematext detects a known service on the monitored host, such as [Nginx](/docs/integration/nginx-integration/) or [Apache](/docs/integration/apache-integration/), it will also suggest creating a service-specific Logs App. These come with out-of-the-box dashboards and alerts tailored to that service. Connecting both gives you application-level events from OTel logs and infrastructure-level events from the service logs in one place. | ||
|
|
||
| ## How Logs Are Matched | ||
|
|
||
| ### With a Trace ID | ||
|
|
||
| This applies to **OpenTelemetry Logs Apps** and is the recommended way to correlate logs. If your backend includes the active trace ID in its HTTP response headers, Sematext reads it from the monitor run result and uses it to filter logs to that exact request. See [Expose Trace ID in Response Headers](/docs/synthetics/root-cause-discovery/adding-trace-id-to-response-headers/) to set this up. | ||
|
|
||
|  | ||
|
|
||
| This eliminates noise from unrelated requests that happened around the same time and takes you directly to the logs for the specific request that triggered the monitor failure. | ||
|
|
||
| ### Without a Trace ID | ||
|
|
||
| If your backend does not include a trace ID in its response headers, logs are matched by the monitored URL and the time window around the failure. The tab shows the number of matching logs — open them in a new tab by clicking the Logs App link. | ||
|
|
||
|  | ||
|
|
||
| ## Exploring Logs | ||
|
|
||
| Once you open the Logs App from this tab, filters are applied automatically — the URL and time range, or the trace ID if available. | ||
|
|
||
| ## Further Reading | ||
|
|
||
| - [Expose Trace ID in Response Headers](/docs/synthetics/root-cause-discovery/adding-trace-id-to-response-headers/) | ||
| - [OpenTelemetry Logs integration](/docs/integration/opentelemetry-logs/) | ||
| - [OpenTelemetry SDKs](/docs/tracing/sdks/) | ||
| - [Logs Discovery](/docs/logs/discovery/intro/) | ||
| - [Context View](/docs/logs/context-view/) |
33 changes: 33 additions & 0 deletions
33
docs/synthetics/root-cause-discovery/metrics-correlation.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| title: Metrics Correlation | ||
| description: Correlate failed Synthetics monitor runs with infrastructure and service metrics. | ||
|
|
||
| The Metrics tab in the failed run flyout lists all Monitoring Apps [connected](/docs/guide/connected-apps/) to your Synthetics App. It gives you a direct path to the metrics of the services hosting your monitored endpoint, scoped to the time of the failure. | ||
|
|
||
| ## Connecting a Monitoring App | ||
|
|
||
| If you don't have a Monitoring App yet, the Metrics section in the Troubleshoot tab will prompt you to create one and connect it to your Synthetics App automatically. If you already have Monitoring Apps in your account, you can select and connect the relevant one directly from the same tab. | ||
|
|
||
| We recommend using the [OpenTelemetry Metrics integration](/docs/integration/opentelemetry-monitoring/) — it produces structured, enriched telemetry that correlates well across signals and gives you more context when investigating failures. | ||
|
|
||
| If Sematext detects a known service on the monitored host, such as [Nginx](/docs/integration/nginx-integration/) or [Apache](/docs/integration/apache-integration/), it will also suggest creating a service-specific Monitoring App. These come with out-of-the-box dashboards and alerts tailored to that service, so you can start monitoring what matters immediately after installing the [Sematext Agent](/docs/agents/sematext-agent/) on your host. | ||
|
|
||
| ## Using the Metrics Tab | ||
|
|
||
| Once a Monitoring App is connected, it appears in the Metrics tab when a monitor run fails. | ||
|
|
||
|  | ||
|
|
||
| Click the App name to open it in a new tab. The Monitoring App opens with the time range pre-set to around the time of the failure. From there, check for anomalies that coincide with the failure: | ||
|
|
||
| - CPU and memory spikes | ||
| - Elevated error rates or dropped request counts | ||
| - Database connection pool exhaustion or high query latency | ||
| - Network throughput drops | ||
|
|
||
| ## Further Reading | ||
|
|
||
| - [Monitoring overview](/docs/monitoring/) | ||
| - [OpenTelemetry Metrics integration](/docs/integration/opentelemetry-monitoring/) | ||
| - [Nginx integration](/docs/integration/nginx-integration/) | ||
| - [Apache integration](/docs/integration/apache-integration/) | ||
| - [Available integrations](/docs/integration/) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.