feat(logs): System logs, log rules, and AI triage - #206
Merged
Conversation
Formatting only. No behaviour changes.
Reading or following a deployment's logs can again be narrowed to a single container instead of always returning every service interleaved. The requested name is checked against the deployment's own compose services first, so an unknown one is rejected with the list of what is available rather than reaching the container runtime. A file log source stays whole, since it is one file the deployment writes rather than per service output.
The reverse proxy's access and error logs are now readable and followable from the panel, so a request that never reached a deployment is visible instead of being invisible everywhere. Access lines now start with the host they were served for, which makes the shared log filterable down to a single deployment by matching its domains. Access and error are offered separately because they are separate outputs, and reading them needs infrastructure access rather than access to any one deployment.
A log can now be emptied from the panel, which frees the disk it was using. A file source is emptied in place so the application keeps writing to the same file, and container output is emptied where the container runtime stores it. A source the runtime does not keep on disk is refused with the reason rather than appearing to succeed. Emptying a deployment's log needs write access to that deployment, and emptying a system log needs infrastructure write access, since neither is recoverable.
Logs can now be watched for patterns that matter, so a repeated error raises an incident on its own instead of waiting for someone to be looking at the right screen at the right moment. A rule sets the level and pattern that count, how many times within how long, and how long to stay quiet after firing. Only an incident reaches the assistant, and only once for each distinct fault: identical lines are collapsed by shape before anything is counted, the context handed over is capped, and a daily ceiling bounds the spend. Incidents carry a short explanation and suggested next step where the assistant is configured, and the plain incident where it is not. Acting on an incident goes through a registry, so notifying is the first responder rather than the only possible one.
Code Review SummaryThis PR introduces a robust system for log management, including the ability to stream and filter proxy access logs per deployment, an automated log rule engine for incident detection, and an AI-powered triage system with built-in cost controls. 🚀 Key Improvements
💡 Minor Suggestions
|
![sourceant[bot]](https://avatars.githubusercontent.com/in/1161781?s=60&v=4){kind=small}
Finding where Docker stores a container's output no longer spawns a CLI process for it, and the call sits with the rest of the Docker work rather than in the HTTP layer. A triage prompt that has to be shortened is now cut on a character boundary, so a log line in any language cannot end in half a character and leave the prompt invalid.
Asking to empty the logs of a service that does not exist answered with success and a count of zero. It now fails with the names that do exist, the same way reading them does.
The access-log test asked for a page the moment the container was created, before nginx had finished starting, so it failed or passed depending on the timing of the machine it ran on.
…ained Raising an incident used to stop the reader that raised it: the model call ran on the same goroutine, so a deployment writing logs went unread for as long as the answer took, up to forty-five seconds. A busy app could fall far behind exactly when something was going wrong with it. Detection now ends at the incident. Explaining it and acting on it happen away from the reader, one at a time, so a burst of incidents cannot become a burst of model calls either. An incident raised while the queue is backed up is still recorded and still shown; it simply arrives without an explanation. A line is now placed in time by when it was written rather than when it was read, so a reconnect no longer squeezes a spread-out burst into an instant and trips a rule that nothing tripped. A timestamp far from the present is treated as a wrong clock and ignored. Asking the internal stream for a source that cannot be read now fails with the reason instead of answering success and then falling silent.
…onger needs The engine kept a little state per stream it had ever seen and per distinct shape of message it had ever matched, and never let either go. On an agent that runs for months, every container that ever wrote a line and every variation of an error stayed in memory for the life of the process. Both are now dropped once they have stopped deciding anything: a burst window after its own window and cooldown have passed, and a stream's recent lines after it has been quiet for an hour. A stream that starts writing again simply builds its context back up.
| return func(line string) bool { | ||
| // Lines arrive with docker's timestamp ahead of nginx's own fields, so the host is | ||
| // one of the first two fields depending on whether that prefix is present. | ||
| fields := strings.Fields(line) |
There was a problem hiding this comment.
Using strings.Fields(line) on every access log line is inefficient because it allocates a slice for the entire line, including potentially long query strings, user agents, or referers. Since the proxy log format leads with the host (and optionally a Docker timestamp), you only need to check the first two whitespace-separated fields. A manual search for the first few spaces would avoid significant allocation overhead in high-traffic environments.
Suggested change
| fields := strings.Fields(line) | |
| // Avoid splitting the entire line; we only care about the host which is | |
| // in the first or second field (depending on the presence of a timestamp). | |
| start := 0 | |
| for f := 0; f < 2; f++ { | |
| for start < len(line) && (line[start] == ' ' || line[start] == '\t') { | |
| start++ | |
| } | |
| if start >= len(line) { | |
| break | |
| } | |
| end := start | |
| for end < len(line) && line[end] != ' ' && line[end] != '\t' { | |
| end++ | |
| } | |
| host := strings.ToLower(line[start:end]) | |
| host = strings.TrimSuffix(host, ":443") | |
| host = strings.TrimSuffix(host, ":80") | |
| if _, ok := hosts[host]; ok { | |
| return true | |
| } | |
| start = end | |
| } |
| return | ||
| } | ||
|
|
||
| kept := make([]string, 0) |
There was a problem hiding this comment.
Using strings.Split on the entire log block can be memory-intensive if the log is large, as it creates a large slice of string pointers. Using a bufio.Scanner with a strings.NewReader would be more memory-efficient here.
Suggested change
| kept := make([]string, 0) | |
| kept := make([]string, 0) | |
| scanner := bufio.NewScanner(strings.NewReader(logs)) | |
| for scanner.Scan() { | |
| line := scanner.Text() | |
| if line != "" && keep(line) { | |
| kept = append(kept, line) | |
| } | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The panel could not show the proxy's own logs, so a request that never reached a deployment was
invisible everywhere, and the access log could not say which site a line belonged to. Access lines
now lead with the host, which is what makes one shared log readable per deployment.
Log triage is bounded on purpose. Lines are collapsed by shape before anything is counted, so a
thousand repeats of one fault are one incident; only an incident reaches the assistant, the
context handed over is capped, and a daily ceiling bounds the spend. Acting on an incident goes
through a registry so notifying is the first responder rather than the only possible one.
Deleting a log truncates in place rather than unlinking, so the application keeps writing to the
same file. A source the runtime does not keep on disk is refused with the reason instead of
appearing to succeed.
The service filter was a regression: it had been dropped from the compose call.
Needs the matching ui branch.
internal/dockertests time out onTestPullDeployment, whichpulls images over the network and does the same on main.