[Improvement-18459][Common] Cap whole-file log download at 47MB to prevent OOM

[xmg333]

Was this PR generated or assisted by AI?

YES. Implementation and tests drafted with assistance from Claude (Anthropic); reviewed by human.

Purpose of the pull request

getFileContentBytesFromLocal read entire files into memory with no size limit. Downloading a large task log caused OOM on the worker.
This PR caps the read at 47 MB and returns a clear error for oversized logs.

Why 47 MB, not 64 MB? The byte[] is JSON-serialized as base64 (~1.33× expansion) before RPC transmission. 47 MB raw → ~63 MB JSON body, staying under the 64 MB maxFrameSize in TransporterDecoder. 64 MB raw would produce ~86 MB body and be rejected by TooLongFrameException.

close #18459

Brief change log

• LogUtils: add MAX_LOG_DOWNLOAD_SIZE = 47 MB; getFileContentBytesFromLocal stops reading once the limit is reached.
• LogServiceImpl: checks file size before reading; returns ERROR with a clear message for oversized logs instead of silently truncating.

Verify this pull request

This change added tests and can be verified as follows:

• LogServiceImplTest: a 48 MB file returns ERROR with message containing "exceeds maximum download size".
• ./mvnw spotless:check passes.

Pull Request Notice

Pull Request Notice

[SbloodyS]

For a log larger than 47 MB:

1. LogServiceImpl#getTaskInstanceWholeLogFileBytes returns ERROR.
2. LogClientDelegate#getWholeLogBytes treats every local error as a reason to call remoteLogClient.getWholeLog(...).
3. RemoteLogClient calls getFileContentBytesFromRemote, which now uses the same capped reader and silently returns only the first 47 MB.

With remote logging enabled, the API can therefore return a successfully downloaded but truncated log. With remote logging disabled or unavailable, it may return an empty log or a generic download error instead of the clear size-limit message.

Please distinguish “local log unavailable” from “log exceeds the supported size,” propagate the latter to the API, and make the reader fail explicitly rather than silently truncating. The regression test should cover the complete LogClientDelegate/API path, not only LogServiceImpl.

Additionally, the linked issue expects large logs to remain downloadable through chunked streaming. This PR rejects them entirely.

[xmg333]

Thanks for the review @SbloodyS . I've reworked the approach based on your feedback. New ** chunked streaming log ** is now completed. Could you confirm if this scope is what you had in mind?

What's new

New RPC: getTaskInstanceLogFileChunk(path, offset, length) reads an 8 MB range via RandomAccessFile. The API loops this RPC and streams
each chunk to the HTTP response via StreamingResponseBody.

Fallback logic (the important part)

The API server runs a chunked loop with an offset counter tracking bytes already written to the response:

streamWholeLog(taskInstance, outputStream):
    if worker not in registry:
        → remote getWholeLogBytes (legacy, unchanged)

    offset = 0
    loop:
        try:
            chunk = localLogClient.getLogChunk(offset, 8MB)
            if chunk.code != SUCCESS:
                if offset == 0:  ← nothing written yet, safe to fallback
                    → remote getWholeLogBytes; return
                else:            ← bytes already streamed, can't restart
                    → throw IOException
            write chunk.bytes; offset += chunk.bytes.length
            if chunk.eof: return
        catch Exception:          ← old worker (method not found), timeout, etc.
            if offset == 0:      ← still safe
                → remote getWholeLogBytes; return
            else:
                → throw IOException

The core invariant: fallback only happens when offset == 0 (nothing written yet). Once bytes have been streamed (offset > 0), there's no
safe way to restart — falling back to getWholeLogBytes would write the whole file from the beginning, duplicating the prefix that's already in
the response. So mid-stream failures throw instead.

Three concrete scenarios:

Scenario	offset	Behavior
Old worker, first chunk RPC fails (method not found)	0	→ fallback to legacy getWholeLogBytes (rolling upgrade safe)
Worker dies mid-stream after writing 24 MB	24 MB	→ throw IOException (client sees truncated download, not corrupted)
Worker offline from the start	0	→ go directly to remote getWholeLogBytes

Legacy path unchanged: getTaskInstanceWholeLogFileBytes and getFileContentBytesFromLocal are untouched from upstream/dev — no silent
truncation, no size cap added.

readFileRange (the new reader) fails explicitly: missing file → IOException, not empty bytes.

Tests cover all three scenarios in LogClientDelegateTest.

[det101]

If remote returns null/empty (archive missing), this still flush()es and the download ends as HTTP 200 with only the log header. Please throw when bytes are absent so a missing remote log is not reported as a successful download.

[xmg333]

Thanks for catching this. Fixed in LogClientDelegate.writeRemoteLegacy: when remoteLogClient.getWholeLog(...) returns null/empty (remote archive missing), it now throws IOException instead of flushing an empty body. The exception propagates through streamWholeLog → StreamingResponseBody and aborts the response, so a missing log is no longer reported as a successful HTTP 200 download.
While reviewing the fix I also found and fixed a related gap: LoggerServiceImpl.checkDownloadLogAuth validated host but not logPath. It's now fixed and would return a clear error before streaming starts.