[Fix-18632][Storage-OSS] Paginate listStorageEntity instead of returning only the first page

[nanxiuzi]

Was this PR generated or assisted by AI?

YES - AI assisted. The issue analysis, the fix proposal and the code change were drafted with AI assistance. I reviewed the change, built the module, deployed the resulting plugin jar to a DolphinScheduler 3.4.2 deployment using OSS as the resource backend, and verified the behaviour manually.

Purpose of the pull request

Close #18632.

When the resource center uses the OSS storage backend, OssStorageOperator#listStorageEntity issued a single listObjectsV2 call with no pagination. The OSS server default of 100 max-keys therefore applied, and because Contents (objects) and CommonPrefixes (directories) share that one quota, any directory with more than 100 combined entries was silently truncated in the UI. Widening the page cannot fix this on its own: max-keys is capped at 1000 by the service.

Brief change log

• OssStorageOperator#listStorageEntity now loops on IsTruncated / NextContinuationToken until all pages are fetched, setting max-keys to 1000 per request (MAX_KEYS constant).
• Common prefixes are deduplicated with a Set that persists across pages, so directory marker objects are still filtered out when they appear on a later page than their common prefix.
• The current directory itself and directory markers already present in CommonPrefixes continue to be excluded from the object listing.

Verify this pull request

This change added no automated tests, but was verified manually as follows:

• mvn -pl dolphinscheduler-storage-plugin/dolphinscheduler-storage-oss -am -DskipTests install builds successfully, and spotless:apply leaves the file unchanged.
• Disassembling the compiled class confirms withMaxKeys, setContinuationToken, isTruncated and getNextContinuationToken are all present in listStorageEntity.
• Verified against a DolphinScheduler 3.4.2 deployment backed by a real Aliyun OSS bucket: the built dolphinscheduler-storage-oss plugin jar was swapped into the deployment and the API server restarted. A resource directory containing more than 100 mixed files and sub-directories now lists every entry in the resource center, where before only the first 100 were returned. Nested browsing via listFileStorageEntityRecursively is likewise complete.

dolphinscheduler-storage-oss currently has no src/test directory, so an automated test would need the Aliyun SDK mocked or a MinIO-style OSS-compatible endpoint. Happy to add one if maintainers would like it in this PR.

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

[nanxiuzi]

Verified in production on a 3.4.2 deployment:

• Built the dolphinscheduler-storage-oss plugin jar from this branch and swapped it into the deployment, then restarted the API server.
• A resource directory containing more than 100 combined files and sub-directories now lists every entry in the resource center. Before the change only the first 100 were returned, matching the OSS server default of max-keys=100.
• Nested browsing (which goes through listFileStorageEntityRecursively) is complete as well.

The change is confirmed working against a real OSS bucket.

[SbloodyS]

LGTM