Feat/dynamic frontend#8
Open
Ironboxplus wants to merge 32 commits into
Open
Conversation
|
如果跨多个组件，请使用主要组件作为前缀，并在标题中枚举、描述中说明。 如果是破坏性变更，请在类型后添加 |
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
6 times, most recently
from
3b720cc to
f0fc91a
Compare
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
2 times, most recently
from
66b9ff7 to
bc14381
Compare
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
from
bc14381 to
0bf12c5
Compare
…m#2471) * fix(driver): fix 189 & 189pc fastcopy form local storage * fix(driver): fix 189 & 189pc fastcopy form local storage * fix(driver): fix 189 & 189pc fastcopy form local storage
Optimize the 189pc driver to implement AccessToken login
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
2 times, most recently
from
02e4430 to
acad358
Compare
* add LinearMemory * replace mmap with LinearMemory * remove unused code * add GuardedMemory; add `min_free_memoryMB` conf * add HybridCache and StreamBuffer * log * rename SizedReadWriterAt to Section * 重构 FileStream，改用 HybridCache * 重构 HybridCache，更新方法名并添加回滚功能；优化请求和流处理逻辑 * 重构 StreamSectionReader 接口，使用HybridCache * 在 NewGuardedMemory 函数中添加了对 LinearMemory 的最终化处理，以确保内存释放 * . * 优化检查逻辑 * 重命名 * 改进、重命名 * 修复 * 添加测试 * 移除HybridCacheReader并引入DynamicReadAtSeeker * 重构缓存读取逻辑，简化代码并引入ReadFromN方法 * 优化缓存配置注释并修复下载器部分大小限制逻辑 * 优化中断逻辑 * 优化下载器代码 * 修复bug * HybridCache添加多文件缓存模式 * 优化下载器并发，添加测试 * 修复bug * 重命名+注释 * . * fix(net): always cleanup downloader on interrupt * fix(net): guard chunk enqueue with context cancel * fix(net): update interrupt logic and add download interrupt test * fix(test): update concurrency limit in high concurrency test * refactor(buffer): simplify ReadAt logic * refactor(config): update memory configuration logic * . --------- Co-authored-by: Suyunmeng <Susus0175@proton.me>
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
from
acad358 to
7995dbd
Compare
…#2478) fix(139): remove RFC8441-incompatible Connection header Agent-Logs-Url: https://github.com/OpenListTeam/OpenList/sessions/1c6b226d-02c4-4b43-80ad-5ab2861d30c3 Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
* refactor(HybridCache)!: extract hybrid_cache package and rename cache_threshold to auto_memory_limit * split HybridCache and stores into internal/hybrid_cache package * introduce BackingStore abstraction with BufferStore and FileStore * rename CacheThreshold to AutoMemoryLimit in config/env/bootstrap * update stream/request integration and related tests * rename singleFileCache and MultiFileCache to singleFileStore and MultiFileStore * refactor(HybridCache): improve memory management strategies in NewHybridCache * rename * alias hybrid_cache to hcache * omments
…returning HTTP 204 No Content on successful authentication (OpenListTeam#2476)
…nd upload hash rework - Add link refresh capability with RefreshableRangeReader for expired download links - Implement self-healing reader for interrupted stream reconnection - Add seekable prefetch window with strict tests - Enhance hash calculation and upload logic for various stream types - Fix link expiry detection to avoid treating context cancellation as expired - Optimize link cache logic, remove SyncClosers dependency - Support all 4xx client errors in expired link check - Fix local file 'file already closed' error - Unify hash calculation progress weight to 100 - Add slow network support with adjusted timeout and retry
…and MD5 checksum - Add duplicate filename handling to ensure file name uniqueness - Add folder creation lock mechanism and retry logic - Increase MakeDir wait time for Google Drive API sync delay - Add retry mechanism for small file upload read errors - Update Put method to support MD5 checksum on seekable and non-seekable streams
…d error handling - Add permanent delete feature with recycle-bin retry logic - Expose proxy_range option in driver meta - Fix duplicate link error handling via error code 10008 detection - Fix folder delete cid JSON number deserialization error - Move offline task Remove before Transfer - Cleanup completed offline download tasks
… mechanism - Optimize task list retrieval with multi-page support and status hints - Add task limit wait mechanism to optimize offline download task processing
… etag fix - Implement streaming upload for Baidu Netdisk - Add rate limiting and retry logic for quark_open, optimize upload handling - Fix quark shard size adjustment for oversized errors - Fix file copy failure to 123pan due to incorrect etag - Fix aliyundrive upload hash calculation
…0 panic - Fix pre-create subdirectory depth from 2 to 1 to avoid deep recursion - Fix srcBasePath bug and remove redundant sleep in preCreateDirectoryTree - Add unit tests for preCreateDirTreeFn - Update storage retrieval method in alias listRoot function - Fix sftp symlink path resolution - Fix 500 panic and NaN issues in openlist driver
…rastructure - Add dynamic frontend pull from GitHub rolling release at startup - Implement frontend caching mechanism with ETag-based validation - Track rolling release by tag commit hash - Unify configurable frontend repo and release channel across build and runtime - Upgrade GitHub Actions to Node24-compatible versions (checkout v5, setup-go v6, etc.) - Update Docker workflow to support frontend version matrix builds - Optimize CI build speed with smart caching
- Add CLAUDE.md for project guidance and development instructions - Add compatibility report - Use Ironboxplus/115-sdk-go v0.2.5 fork in go.mod replace directive
…imit, dist swap lock, Go 1.26 vet - Google Drive Put: replace infinite 401 recursion with maxPutAuthRetries=2 - Google Drive MakeDir: delete mkdirLocks entry after unlock to prevent memory leak - selfHealingReadCloser: only reconnect on io.ErrUnexpectedEOF, not normal io.EOF - RefreshableRangeReader: document concurrency contract for innerReader replacement - Frontend extractTarGz: add maxExtractFileSize (500MB) and io.LimitReader - Frontend dist swap: add distSwapMu to protect rename window - Fix Go 1.26 non-constant format string vet errors across drivers
added 10 commits
May 19, 2026 18:11
- NewOSSUploadHttpClient: ResponseHeaderTimeout 15s → 5min for uploads - Handle PartAlreadyExist (409) in retry by recovering part via ListUploadedParts instead of failing, fixes timeout-then-retry conflict with oss.Sequential()
Put() only called WaitLimit once at entry, but then fired 3-4 SDK requests (UploadInit ×3, UploadGetToken) without rate limiting. Under concurrent copy tasks this overwhelmed the 115 API, causing state:false/code:0 rejections on all requests including List/MakeDir. Now each SDK call in Put is individually rate-limited, matching the pattern used by List, MakeDir, Move, etc.
The SDK's authRequest had no concurrency protection on token refresh. Multiple goroutines hitting 401 simultaneously would all call RefreshToken independently, causing "refresh frequently" errors and corrupting the stored tokens. v0.2.6 adds mutex with double-check pattern to ensure only one goroutine refreshes at a time.
115 API returns {state:true, data:[]} for non-existent paths instead of
an error code. SDK v0.2.8 adds ErrDataEmpty sentinel; driver.Get() maps
it to ObjectNotFound so MakeDir can create directories normally.
…iter hash truncation
Three bugs fixed:
1. Upload callback was never validated — when 115 returned {"state":false},
the error was silently ignored. Added oss.CallbackResult capture and
checkUploadCallback() validation.
2. CacheFullAndWriter for known-size files > MaxBufferLimit only cached the
first ~48MB (via cache() truncation), producing a partial SHA1 hash.
115 then rejected the upload with code=10002 "校验文件失败". Fix: when
file size exceeds MaxBufferLimit, write to a temp file instead of the
in-memory peekBuff. This fixes all drivers using CacheFullAndHash with
large non-seekable streams (115, google_drive, aliyundrive, etc).
3. Frontend fetcher blocked startup on China servers where GitHub is
unreachable. Removed blocking download, use embedded dist immediately
and let Watcher fetch rolling release in background.
Overlap source-side I/O with hash/upload compute in both passes of an upload: - Pass 1 (hash): streamHashSeekableWithPrefetch keeps a background goroutine fetching the next ~10MB chunk via ReadFullWithRangeRead while the current chunk feeds the hash. Identical hash output, only the timing changes from `Tr + Th` per chunk to `max(Tr, Th)`. - Pass 2 (upload): hybridSectionReader.GetSectionReader schedules a background prefetch of the next block while the caller uploads the current one. takePrefetched / waitPrefetch handle hits, misses, DiscardSection, and clean shutdown via file.Add(closerFunc(...)). Effect: drivers that loop sequentially without their own Before/Do split (115_open with oss.Sequential(), baidu_netdisk, aliyundrive_open, etc.) see throughput close to single-side bandwidth instead of the sum of read+write latencies.
Removed preCreateDirTreeFn and its method wrapper. Directory creation
is no longer batched up-front via a one-level BFS walk; instead it
happens on demand:
- top-level dst MakeDir still runs once at line 198 (early failure
detection, active dir creation)
- sub-task RunWithNextTaskCallback creates each subdir as it
recurses into it
- op.Put internally calls MakeDir(parent) before uploading
- op.MakeDir walks the parent chain so deep trees work without
pre-creation, and our cache-sync retry handles cloud-storage
eventual consistency
The merge-mode existedObjs logic (the only invariant the BFS
precreate quietly protected, alongside OpenListTeam#1898's ObjectNotFound
tolerance) is extracted into a pure, injectable function:
existingDstFilesFn(ctx, listDst, dstPath) (map[string]bool, error)
12 unit tests pin down its contract: empty dst, mixed files/dirs,
raw and wrapped ObjectNotFound (regression for OpenListTeam#1898), other List
errors, ctx cancellation before/during iteration, duplicates,
large dst, single-call sanity. The old 11 TestPreCreateDirTreeFn_*
tests are removed with the code.
…te removal JOURNAL.md: new entry covering the three coordinated changes (commits fccab33 → 7995dbd → 6b3ce57) — rebase onto upstream HybridCache, Pass 2 prefetch on hybridSectionReader, and BFS precreate deletion. CLAUDE.md: replace the now-obsolete "MaxBufferLimit truncation pitfall" section with the current HybridCache story; add new sections for the hybridSectionReader prefetch and the merge-task ObjectNotFound tolerance contract (incl. the existingDstFilesFn extraction).
The merge skip rule at copy_move.go:221 has two halves whose
combination is load-bearing:
1. existingDstFilesFn must only include FILES (not dirs) from dst
2. the call-site guard `!obj.IsDir() && existedObjs[name]` must
keep the !obj.IsDir() clause
If either half drifts (e.g. dirs leak into existedObjs, or the
IsDir guard is removed), src subdirectories matching dst dirs of
the same name would be silently skipped — every file inside them
would never get copied. That bug is invisible to "did the task
complete?" checks; only shows up as quietly missing deep files.
New tests:
- TestMergeSkipDecision_DirsAreNeverSkipped: 7-case matrix
covering every src kind × dst state combo, including the
cross-type name collisions (src dir / dst file and vice versa)
- TestMergeSkipDecision_EmptyDst: nothing skipped against empty dst
- TestMergeSkipDecision_DeepTreeRecursionContract: 3-level src
tree against partial dst, asserts correct skip/spawn at each
level — the canonical "deep files missing" regression guard
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
from
7e14090 to
39c7467
Compare
added 5 commits
May 19, 2026 21:18
115 CDN signed URLs carry a Unix-timestamp `t=...` query parameter that marks when the URL stops accepting requests — typically just a few minutes, well below OpenList's default link-cache TTL (30 min). When the cache outlives the URL, OP serves a stale URL and 115 responds with HTTP 200 + Content-Length: 0, which clients (mpv, ffmpeg, …) see as a corrupt or empty stream. Since 115's link cache is keyed per User-Agent (`LinkCacheMode: LinkCacheUA`), the failure is sticky and per-UA: the "libmpv" entry stays poisoned while other UAs keep working through the webpage, masking the bug as "only mpv fails". Parse the `t=` parameter and set `model.Link.Expiration` so OP refreshes the link before the CDN actually rejects. A 60-second safety margin covers clock skew and in-flight requests; any timestamp at-or-near expiry clamps to a 1-second minimum so the cache entry evicts on next lookup instead of being held indefinitely. 10 unit tests pin the helper contract (future / past / about-to-expire / missing / malformed / empty / negative / invalid-URL / real-world / safety-margin).
When a 115 CDN signed URL expires, the server answers any request with
HTTP 200 + Content-Length: 0 (a "soft success") rather than 4xx. The
existing selfHealingReadCloser catches this AFTER body read returns
(0, io.EOF) and triggers a refresh, but only when a Refresher is set
and ServeHTTP has already written response headers — the client gets
HTTP 206 + Content-Length: ${expected} headers immediately followed by
zero bytes, breaking range-aware clients that pre-allocate buffers.
Move the detection up to the request level: when the closure in
GetRangeReaderFromLink asks for ≥1 byte but the response promises 0,
return an "expired" error before ever exposing the empty body. The
RefreshableRangeReader path now refreshes one step earlier, and callers
without a Refresher fail loudly instead of streaming silence.
ContentLength == -1 (chunked transfer) is excluded so legitimately
chunked responses are not flagged.
4 tests cover:
- soft-expired + Refresher → refresh fires once, fresh body delivered
- soft-expired + no Refresher → IsLinkExpiredError-classified error
- healthy 206 → no false positive
- chunked transfer → no false positive
The libmpv "Failed to recognize file format" symptom came from one
underlying mistake: 115's GetFolderInfoByPath endpoint (yuque
rl8zrhe2nag21dfw, "Get folder details") is folder-oriented. Its struct
carries Count / FolderCount / PlayLong and the Size field is
byte-accurate only for directories; for FILE paths Size is empty or
"0". driver.Get nevertheless called it for every path. The resulting
Obj had FS = 0, op.Get returned it to ServeHTTP, and the empty-file +
Range branch in internal/net/serve.go answered with 200 OK +
Content-Length: 0. mpv couldn't recognize the empty stream.
Webpage playback masked the bug because op.List warmed dirCache with
GetFilesResp_File entries (FS arrives as int64 there and is always
correct). mpv direct access skipped op.List, hit driver.Get cold, and
broke.
Fix:
- Introduce fromFolderInfo gateway used by driver.Get:
* folder → return Obj from folderInfoToObj (fast path, single API call)
* file → return errs.NotImplement so op.Get falls through to its
list-based path, which pulls FS out of
GetFilesResp_File.FS (int64, reliable)
- folderInfoToObj extracts only the fields the gateway actually uses
(FileID, FileName, FileCategory, Sha1, PickCode). Size is not parsed —
files are rejected before any caller would read FS.
- log.Debugf in driver.Get prints the raw resp.Size / FileCategory so
future diagnoses don't need to theorize.
Trade-off: cold-cache file access costs 3 WaitLimit-gated SDK calls
(wasted GetFolderInfoByPath on the file + GetFolderInfoByPath on the
parent for its FileID + GetFiles to list the parent) plus the eventual
DownURL = 4 calls. With default limit_rate=1 req/s that's ~3s of pure
rate-limit wait. Steady-state access within dirCache TTL (5 min)
collapses back to a single DownURL. Bumping limit_rate to 5–10 in the
storage config keeps the worst case under 1s.
Verified in production with curl + libmpv UA + Range: bytes=0-1023 →
206 Partial Content / Content-Length: 1024 / Content-Range bytes
0-1023/36767958354 / Last-Modified populated (proving the Obj came
from the List path, not the bogus folderInfoToObj of an earlier
attempt).
5 unit tests pin the contract: folder fast path, file/folder gateway
behavior, and an existing TestGetReturnsObjForExistingFolder updated
to set file_category="0" so it actually exercises the folder branch.
… names Two changes that go together because the second was useless without the first. Push to feat/dynamic-frontend (the active branch) was not in the trigger list, so commits like Fix 3 b80688c never produced a fresh :front-rolling image -- the user kept pulling stale binaries and the mpv "Failed to recognize file format" symptom appeared unfixed when in fact CI had simply not run. - Replace push:main + pull_request:copy triggers with push:feat/dynamic-frontend so every commit on the active branch rebuilds the four docker variants. - Drop the shared IMAGE_NAME env and give each matrix row its own image_name: openlist, openlist-ffmpeg, openlist-aria2, openlist-aio. Tags collapse from openlist:front-rolling-{variant} to {image_name}:front-rolling, so the pull path matches the image intent (ghcr.io/.../openlist-aio:front-rolling, etc.). - :latest now applies to each image when its frontend_channel is "latest", rather than only openlist:latest. - Build cache key reuses the per-variant image_name so caches don't collide across variants.
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
from
3c908da to
d569888
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description / 描述
Motivation and Context / 背景
Closes #XXXX
Relates to #XXXX
How Has This Been Tested? / 测试
Checklist / 检查清单
我已阅读 CONTRIBUTING 文档。
go fmtor prettier.我已使用
go fmt或 prettier 格式化提交的代码。我已为此 PR 添加了适当的标签（如无权限或需要的标签不存在，请在描述中说明，管理员将后续处理）。
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
我已相应更新了相关仓库（若适用）。