ADLS Gen 2#2635
Open
Skrypt wants to merge 34 commits into
Open
Conversation
Adds ADSL Gen 2 feature parity to Azurite. See docs/design/ADLS-gen2-parity.md file for details.
Author
|
@microsoft-github-policy-service agree |
Author
|
hmm ok, just found this doc here: https://github.com/Azure/Azurite/wiki/ADLS-Gen2-Implementation-Guidance |
…DFS endpoint Close compliance gaps with the Azurite ADLS Gen2 Implementation Guidance wiki: - Add --enableHierarchicalNamespace CLI flag (default: true) so the emulator can run in FNS or HNS mode; wired through BlobEnvironment, Environment, VSCEnvironment, DfsServer, and FilesystemHandler. - Replace non-atomic copy+delete rename with persistence-level renameBlob() and renameBlobsByPrefix() on both LokiJS and SQL stores, keeping directory renames atomic and avoiding unnecessary extent copies. - Extract and forward If-Match, If-None-Match, If-Modified-Since, and If-Unmodified-Since conditional headers on DFS getProperties, read, and delete operations via ModifiedAccessConditions. - Expose full blob lease lifecycle (acquire, release, renew, break, change) on the DFS endpoint by detecting x-ms-lease-action and delegating to the existing IBlobMetadataStore lease infrastructure. - Add DFS-specific error helpers for conditional and lease failures. - Extend dfsProxy.test.ts with tests for conditional headers, all lease operations, and atomic directory rename with children verification. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduce the Swagger-generated DFS interface layer as called for by the Azurite ADLS Gen2 Implementation Guidance wiki: - swagger/dfs-storage-2023-11-03.json: OpenAPI 2.0 spec covering all DFS REST API operations (filesystem CRUD, path CRUD, read, update, lease). - swagger/dfs.md: AutoRest configuration for DFS code generation. - src/blob/generated-dfs/: Generated-style TypeScript artifacts mirroring the blob generated layer pattern (src/blob/generated/): - artifacts/operation.ts: DfsOperation enum (12 operations) - artifacts/models.ts: Typed request/response interfaces for all DFS ops - artifacts/specifications.ts: Dispatch specs for HTTP-to-operation matching - handlers/IFilesystemHandler.ts, IPathHandler.ts: Handler contracts - handlers/IHandlers.ts: Combined IDfsHandlers interface - handlers/handlerMappers.ts: Operation-to-handler routing map - Context.ts: DFS request context for the generated pattern - package.json: Added build:autorest:dfs script - DfsRequestListenerFactory.ts: Architecture docs referencing generated layer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Implement the dedicated hierarchy table required by the Azurite ADLS Gen2
Implementation Guidance wiki ("Add table matching each item with parent").
- IBlobMetadataStore: Add registerHnsPath, unregisterHnsPath,
unregisterHnsPathsByPrefix, renameHnsPaths, isHnsDirectoryEmpty, and
hnsPathExists methods for managing parent-child relationships.
- LokiBlobMetadataStore: New $HNS_HIERARCHY$ collection indexed on
(accountName, containerName, path) and (parentPath) for fast lookups.
- SqlBlobMetadataStore: New HnsHierarchy table with unique path index and
parentPath index; rename operations use transactions.
- PathHandler: create() registers paths in hierarchy; delete() uses
isHnsDirectoryEmpty() for non-recursive guard and cleans up hierarchy
records; renamePath() updates hierarchy atomically;
ensureIntermediateDirectories() registers intermediate dirs.
- Tests: non-empty directory delete returns 409; recursive delete succeeds;
auto-created intermediate directories are visible via HEAD.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds native ADLS Gen2 (DFS) endpoint support to Azurite by standing up a first-class DFS server that shares the existing blob metadata/extent stores, plus a broad test suite and configuration/documentation updates.
Changes:
- Introduces a native DFS Express pipeline (context/auth/dispatch + filesystem/path handlers) running on a new configurable DFS host/port.
- Extends blob metadata stores with atomic rename operations to support DFS rename semantics (single-path + prefix/directory).
- Adds DFS-focused integration tests and updates CLI/VSC/Docker/README to expose the DFS endpoint and HNS toggle.
Reviewed changes
Copilot reviewed 30 out of 30 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/blob/dfsProxy.test.ts | Adds DFS end-to-end tests for filesystem/path ops, append/flush, ACLs, leases, and rename. |
| src/common/VSCEnvironment.ts | Adds dfsHost/dfsPort and enableHierarchicalNamespace to VS Code environment config. |
| src/common/Telemetry.ts | Adds dfsHost/dfsPort to telemetry parameter collection. |
| src/common/Environment.ts | Adds CLI flags for DFS host/port and hierarchical namespace toggle. |
| src/blob/utils/constants.ts | Adds DFS defaults and changes HNS default constant behavior. |
| src/blob/persistence/SqlBlobMetadataStore.ts | Implements SQL rename operations for DFS (single blob + prefix). |
| src/blob/persistence/LokiBlobMetadataStore.ts | Implements Loki rename operations for DFS (single blob + prefix). |
| src/blob/persistence/IBlobMetadataStore.ts | Extends store interface with rename methods used by DFS rename. |
| src/blob/main.ts | Starts DFS server alongside standalone blob server and wires shared stores. |
| src/blob/dfs/handlers/PathHandler.ts | Implements DFS path CRUD, list paths, append/flush, ACLs, leases, and rename. |
| src/blob/dfs/handlers/FilesystemHandler.ts | Implements DFS filesystem CRUD/list and property setting via container operations. |
| src/blob/dfs/DfsPropertyEncoding.ts | Adds shared base64 properties header encode/decode helper. |
| src/blob/dfs/DfsOperation.ts | Defines DFS operation enum for routing/auth mapping. |
| src/blob/dfs/DfsErrorFactory.ts | Adds DFS JSON error shaping (including special-casing HEAD). |
| src/blob/dfs/DfsContextFactory.ts | Creates minimal blob Context objects for store calls from DFS handlers. |
| src/blob/dfs/DfsContext.ts | Extracts DFS request context (account/filesystem/path) and performs version checks. |
| src/blob/dfs/DfsAuthenticationMiddleware.ts | Adds DFS auth middleware reusing blob authenticators. |
| src/blob/SqlBlobServer.ts | Exposes stores publicly to allow DFS to share them. |
| src/blob/IBlobEnvironment.ts | Extends blob environment interface with DFS host/port + HNS toggle. |
| src/blob/DfsServer.ts | Adds new DFS server implementation (HTTP/HTTPS) using shared stores. |
| src/blob/DfsRequestListenerFactory.ts | Adds DFS Express pipeline and operation dispatch. |
| src/blob/DfsConfiguration.ts | Adds DFS configuration wrapper based on existing ConfigurationBase. |
| src/blob/BlobServer.ts | Exposes stores publicly to allow DFS to share them. |
| src/blob/BlobEnvironment.ts | Adds blob-service CLI flags for DFS host/port and HNS toggle. |
| src/azurite.ts | Starts DFS server alongside blob/queue/table in the main Azurite entrypoint. |
| package.json | Adds VS Code extension config entries for dfsHost/dfsPort. |
| docs/designs/ADLS-gen2-parity.md | Documents the ADLS Gen2 parity approach, phases, and design rationale. |
| README.md | Documents DFS host/port usage and docker port exposure. |
| Dockerfile.Windows | Exposes DFS port and starts Azurite with dfsHost binding. |
| Dockerfile | Exposes DFS port and starts Azurite with dfsHost binding. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add comprehensive integration tests using the official Azure DataLake SDK
to validate Azurite DFS endpoint compatibility, as required by the wiki
("Pass all language SDK tests").
- Install @azure/storage-file-datalake@^12.29.0 as devDependency
- tests/blob/dfsSDKIntegration.test.ts: 22 tests across 6 categories:
- Filesystem: create/delete, getProperties, listFileSystems
- Directory: create/delete, nested dirs, move (rename)
- File: create, append+flush+read, multi-chunk write, delete, move
- ACL: setAccessControl/getAccessControl, setPermissions
- List paths: recursive and non-recursive listing
- Cross-API: DFS file readable via Blob API and vice versa
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implement OAuth with ACL enforcement as specified by the Azurite ADLS Gen2
wiki Phase III ("OAuth: ACL works when user login with AAD account").
- OAuthLevel: Add ACL level (--oauth acl) alongside existing BASIC level.
- ConfigurationBase: Accept "acl" as --oauth parameter value.
- DfsContext: Add IDfsAuthenticatedIdentity interface with oid, upn, tid,
appid fields; add identity field to IDfsContext.
- DfsAuthenticationMiddleware: Extract identity claims (oid, upn, tid,
appid) from Bearer JWT tokens when ACL mode is active; store in DFS
context for downstream enforcement.
- DfsAclEnforcer: New module implementing POSIX ACL evaluation:
- Parses ACL strings ("user::rwx,user:oid:r-x,group::r-x,other::---")
- Evaluates in POSIX order: owner -> named user -> group -> other
- Applies mask entries to limit named user/group permissions
- $superuser and unauthenticated requests bypass checks
- Maps operations to required permissions (r/w/x)
- PathHandler: Add enforceAcl() helper; check ACL before getProperties,
read, delete, and update operations; accept OAuthLevel from factory.
- DfsRequestListenerFactory: Pass OAuth level to PathHandler.
- Tests: 19 unit tests for ACL enforcer covering parsing, bypass scenarios,
owner/named-user/group/other permissions, mask application, and UPN
matching. All passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author
|
I added new commits to comply to the https://github.com/Azure/Azurite/wiki/ADLS-Gen2-Implementation-Guidance. |
The recursive delete was relying on the HNS hierarchy table to check for children, which missed blobs created via the Blob API or before HNS tracking was added. Now always uses listBlobs prefix scan (no delimiter) to find ALL descendant blobs regardless of how they were created, then deletes them all before removing the directory marker itself. Fixes: DeleteDirectory_WithContents_DeletesAll test failure where directoryClient.DeleteAsync(recursive: true) deleted the directory marker but left child blobs intact. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds native ADLS Gen2 (DFS) endpoint support to Azurite by introducing a DFS server pipeline that shares the Blob service’s metadata/extent/account stores, along with HNS directory semantics and initial ACL enforcement.
Changes:
- Introduces a native DFS server (port 10004 by default) with filesystem/path handlers, auth middleware reuse, and DFS-specific context/error handling.
- Extends metadata stores (Loki + SQL) to support rename operations and an HNS hierarchy registry.
- Adds DFS-focused tests (proxy-style REST tests, SDK integration tests, and ACL enforcer unit tests) plus swagger artifacts/docs.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/blob/dfsSDKIntegration.test.ts | Adds JS SDK integration coverage via @azure/storage-file-datalake. |
| tests/blob/dfsProxy.test.ts | Adds direct DFS REST-level tests and cross-API (Blob↔DFS) checks. |
| tests/blob/dfsAclEnforcer.test.ts | Adds unit tests for ACL parsing/evaluation logic. |
| swagger/dfs.md | Adds AutoRest configuration for DFS server generation. |
| swagger/dfs-storage-2023-11-03.json | Adds DFS swagger spec used for generated artifacts. |
| src/common/models.ts | Extends OAuth level enum with ACL enforcement mode. |
| src/common/VSCEnvironment.ts | Adds VS Code settings for DFS host/port + HNS toggle. |
| src/common/Telemetry.ts | Adds dfsHost/dfsPort telemetry parameter handling. |
| src/common/Environment.ts | Adds CLI flags for dfsHost/dfsPort and HNS enablement. |
| src/common/ConfigurationBase.ts | Parses new --oauth acl option. |
| src/blob/utils/constants.ts | Adds DFS defaults + flips HNS “enabled” constant default. |
| src/blob/persistence/SqlBlobMetadataStore.ts | Adds HNS hierarchy table + rename/HNS operations in SQL store. |
| src/blob/persistence/LokiBlobMetadataStore.ts | Adds rename/HNS operations and HNS collection in Loki store. |
| src/blob/persistence/IBlobMetadataStore.ts | Extends store interface with rename + HNS hierarchy methods. |
| src/blob/main.ts | Starts DFS server alongside blob server in azurite-blob. |
| src/blob/generated-dfs/handlers/handlerMappers.ts | Adds generated DFS handler mapper wiring. |
| src/blob/generated-dfs/handlers/IPathHandler.ts | Adds generated DFS path handler interface. |
| src/blob/generated-dfs/handlers/IHandlers.ts | Adds generated DFS handler registry interface. |
| src/blob/generated-dfs/handlers/IFilesystemHandler.ts | Adds generated DFS filesystem handler interface. |
| src/blob/generated-dfs/artifacts/specifications.ts | Adds operation matching specs for DFS routing. |
| src/blob/generated-dfs/artifacts/operation.ts | Adds generated DFS operation enum. |
| src/blob/generated-dfs/artifacts/models.ts | Adds generated DFS request/response model types. |
| src/blob/generated-dfs/Context.ts | Adds DFS-specific generated context wrapper. |
| src/blob/dfs/handlers/PathHandler.ts | Implements core DFS path operations (create/read/delete/list/update/lease/rename). |
| src/blob/dfs/handlers/FilesystemHandler.ts | Implements DFS filesystem ops (create/delete/properties/list/setProperties). |
| src/blob/dfs/DfsPropertyEncoding.ts | Adds helper for DFS x-ms-properties encoding/decoding. |
| src/blob/dfs/DfsOperation.ts | Adds DFS operation enum for middleware/dispatch. |
| src/blob/dfs/DfsErrorFactory.ts | Adds DFS JSON error writer + common DFS error types. |
| src/blob/dfs/DfsContextFactory.ts | Adds helper to create minimal Blob Context for store calls. |
| src/blob/dfs/DfsContext.ts | Adds DFS context middleware and URL parsing. |
| src/blob/dfs/DfsAuthenticationMiddleware.ts | Reuses blob authenticators for DFS + extracts identity for ACL mode. |
| src/blob/dfs/DfsAclEnforcer.ts | Adds ACL parsing and evaluation logic for DFS operations. |
| src/blob/SqlBlobServer.ts | Exposes stores publicly for DFS server sharing. |
| src/blob/IBlobEnvironment.ts | Extends blob environment interface with DFS host/port + HNS toggle. |
| src/blob/DfsServer.ts | Adds DFS server wrapper around request listener pipeline. |
| src/blob/DfsRequestListenerFactory.ts | Adds Express pipeline for DFS routing/auth/handlers. |
| src/blob/DfsConfiguration.ts | Adds configuration class for DFS endpoint. |
| src/blob/BlobServer.ts | Exposes stores publicly for DFS server sharing. |
| src/blob/BlobEnvironment.ts | Adds CLI flags for DFS host/port and HNS toggle in azurite-blob. |
| src/azurite.ts | Starts DFS server alongside blob/queue/table in the combined entrypoint. |
| package.json | Adds @azure/storage-file-datalake dev dependency and DFS extension settings/script. |
| docs/designs/ADLS-gen2-parity.md | Adds design/phase plan for DFS parity implementation. |
| README.md | Documents DFS host/port config and Docker port exposure. |
| Dockerfile.Windows | Exposes 10004 and passes --dfsHost. |
| Dockerfile | Exposes 10004 and passes --dfsHost. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Azure ADLS Gen2 returns 409 when creating a directory that already exists. The SDK's CreateIfNotExistsAsync relies on this to return null/false for existing directories. Azurite was returning 201 regardless, causing CreateDirectory_AlreadyExists_ReturnsFalse to fail. Now checks if a directory blob with hdi_isfolder=true already exists before creating, and returns 409 PathAlreadyExists if so. File creates remain idempotent (overwrite) matching Azure behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author
|
I've updated the unit tests on this PR: OrchardCMS/OrchardCore#19014 |
Guard against Express query params and request body being arrays instead of strings. CodeQL flagged req.query.position and req.body as potential vectors for type confusion in appendData and flushData. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author
|
@copilot open a new pull request to apply changes based on the comments in this thread |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds native ADLS Gen2 (DFS) endpoint support to Azurite by introducing a dedicated DFS server/pipeline that shares the existing Blob metadata + extent stores, plus new persistence helpers (rename + HNS hierarchy) and a broad set of DFS-focused tests and swagger artifacts.
Changes:
- Introduce a DFS server (port 10004 by default) with native Express handlers for filesystem/path operations, append/flush, leases, rename, and ACL metadata.
- Extend metadata stores (Loki + SQL) with rename APIs and HNS hierarchy tracking.
- Add DFS integration/proxy/unit tests, swagger inputs/config, and environment/Docker/README wiring for DFS.
Reviewed changes
Copilot reviewed 45 out of 46 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/blob/dfsSDKIntegration.test.ts | New JS SDK integration tests using @azure/storage-file-datalake. |
| tests/blob/dfsProxy.test.ts | New REST-level DFS behavior tests (create/list/append/flush/rename/ACL/leases). |
| tests/blob/dfsAclEnforcer.test.ts | Unit tests for ACL parsing/evaluation. |
| swagger/dfs.md | AutoRest config for DFS codegen. |
| swagger/dfs-storage-2023-11-03.json | DFS swagger used for codegen/artifacts. |
| src/common/models.ts | Add OAuthLevel.ACL enum value. |
| src/common/VSCEnvironment.ts | Add dfsHost/dfsPort + enableHierarchicalNamespace settings. |
| src/common/Telemetry.ts | Include DFS flags in telemetry parameter list. |
| src/common/Environment.ts | Add --dfsHost/--dfsPort and --enableHierarchicalNamespace CLI flags. |
| src/common/ConfigurationBase.ts | Parse --oauth acl into OAuthLevel.ACL. |
| src/blob/utils/constants.ts | Add DFS defaults; switch emulator HNS default to enabled. |
| src/blob/persistence/SqlBlobMetadataStore.ts | Add HNS hierarchy table + rename helpers. |
| src/blob/persistence/LokiBlobMetadataStore.ts | Add HNS hierarchy collection + rename helpers. |
| src/blob/persistence/IBlobMetadataStore.ts | Add interfaces for rename and HNS hierarchy operations. |
| src/blob/main.ts | Start DFS server alongside blob for azurite-blob. |
| src/blob/generated-dfs/** | New generated DFS artifacts/interfaces/specs. |
| src/blob/dfs/** | New DFS middleware, handlers, ACL enforcer, errors, property encoding. |
| src/blob/SqlBlobServer.ts | Expose stores publicly for DFS server sharing. |
| src/blob/IBlobEnvironment.ts | Add DFS + HNS configuration surface. |
| src/blob/DfsServer.ts | New DFS server wrapper over ServerBase. |
| src/blob/DfsRequestListenerFactory.ts | New DFS Express pipeline (context/dispatch/auth/routing/errors). |
| src/blob/DfsConfiguration.ts | New DFS configuration class. |
| src/blob/BlobServer.ts | Expose stores publicly for DFS server sharing. |
| src/blob/BlobEnvironment.ts | Add DFS host/port + HNS flag for azurite-blob. |
| src/azurite.ts | Start DFS server alongside blob/queue/table for azurite. |
| package.json | Add DFS autorest script + @azure/storage-file-datalake dep + VS Code settings for DFS host/port. |
| docs/designs/ADLS-gen2-parity.md | Design/phase plan for ADLS Gen2 parity implementation. |
| README.md | Document DFS host/port + Docker port mapping updates. |
| Dockerfile.Windows | Expose port 10004 and pass --dfsHost. |
| Dockerfile | Expose port 10004 and pass --dfsHost. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Author
|
@copilot open a new pull request to apply changes based on the comments in this thread |
Author
|
I'm sorry, I probably don't have the permission to run agents on this repository. |
Author
|
No feedback on this one yet after 2 weeks. I know that this is AI generated and this is probably not optimal but at least some feedback would be welcomed. Should I simply drop the PR and consider simply keeping this on my own fork? @sebastienros sorry to ping you on this one. Totally non-related with OC but what's your thoughts on this? This is an open sourced project by Microsoft on the Azure repository. Not sure who's managing this or if I will ever get feedback at this point. Thanks. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 56 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
4 new criticals, 7 majors, 5 minors identified. All pass-1 items marked fixed. 6 new test gaps recorded. Pass-2 items are pending.
…or handling
Criticals (azurite_hns_enabled integrity):
- FilesystemHandler.setProperties: read existing metadata first, preserve HNS flag
and existing user metadata before applying x-ms-properties overlay (P2-C-1, P2-m-3)
- FilesystemHandler.extractMetadata: filter x-ms-meta-azurite_hns_enabled to block forgery (P2-C-4)
- ContainerHandler.setMetadata: preserve azurite_hns_enabled from existing metadata
when Blob API SetContainerMetadata replaces the metadata map (P2-C-2)
- ContainerHandler.getContainerProperties: filter azurite_hns_enabled from user-visible
metadata in Blob API response (P2-C-3)
- FilesystemHandler: add newEtag() import; use it in setProperties
Majors:
- listPaths: return 404 when specified directory does not exist (P2-M-1)
- listPaths: read stored dfsAclOwner/dfsAclGroup/dfsAclPermissions per entry
instead of hardcoded $superuser defaults (P2-M-7)
- PathHandler.delete: handle 412 ConditionNotMet from conditional-header mismatch (P2-M-2)
- DfsContext: replace raw res.status(400).json() with sendDfsError so response
includes x-ms-error-code header (P2-M-3)
- Multi-block read: call stream.destroy() on error to prevent resource leak (P2-M-4)
- breakLease: validate x-ms-lease-break-period for NaN; return 400 if invalid (P2-M-5)
- appendData: document TOCTOU limitation for concurrent appends (P2-M-6)
Minors:
- PathHandler: replace dynamic require("crypto") with top-level import (P2-m-1)
- PathHandler.create: rename inner parentPath to hnsParentPath to avoid shadowing (P2-m-2)
- DfsRequestListenerFactory: restrict resource=filesystem + ctx.path dispatch to GET only (P2-m-4)
- ensureIntermediateDirectories: throw 409 if a file occupies an intermediate path (P2-m-5)
Tests (4 new in dfsProxy.test.ts):
- HNS flag survives filesystem setProperties PATCH
- listPaths returns 404 for non-existent directory
- delete with non-matching If-Match returns 412
- listPaths returns stored ACL owner/group per path
55 DFS @loki tests passing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 56 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
6 majors, 7 minors, 5 test gaps. All pass-2 items marked fixed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 56 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ead code
Majors:
- Recursive delete: pass lease conditions to child blob deletes so leased children
are rejected rather than force-deleted (P3-M-1)
- listPaths: add eTag field to prefix (subdirectory) entries in non-recursive listing (P3-M-2)
- setAccessControlRecursive: only include root directory on first continuation page,
not on every page (inflated directoriesSuccessful count) (P3-M-3)
- setAccessControlRecursive: validate mode param; return 400 InvalidQueryParameterValue
for any value other than set/modify/remove (P3-M-4)
- read: return 400 PathIsDirectory when GET targets a directory blob (P3-M-5)
- renamePathAtomic: re-key uncommitted blocks staged under old path in both
Loki (blocks collection) and SQL (BlocksModel) so flush after rename works (P3-M-6)
Minors:
- Document rename overwrite TOCTOU limitation (P3-m-1)
- Fix invalidFlushPosition() in DfsErrorFactory: status 400→409, parameterised
message; replace inline error object in flushData with the factory call (P3-m-2)
- Single-block read: stream.pipe(res, { end: false }) to prevent double res.end() (P3-m-3)
- FilesystemHandler.list: clamp maxResults with || fallback to prevent NaN (P3-m-4)
- Delete dead DfsPropertyEncoding.ts (never imported) (P3-m-5)
- FilesystemHandler.create: replace custom hex ETag with newEtag() (P3-m-6)
- renamePath: reject rename source paths containing '..' segments (P3-m-7)
Tests (4 new):
- GET on directory path returns 400 PathIsDirectory
- setAccessControlRecursive with invalid mode returns 400
- listPaths non-recursive: subdirectory entries include eTag
- FilesystemHandler.list with non-numeric maxResults does not crash
59 DFS @loki tests passing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@Skrypt Thanks for your work on this. I am not involved with this project but I had occasion to need to do some local testing with DFS support so I tried out your PR. I was using the hadoop-azure (java) layer to interface with Azurite DFS which I ran into these issues (outlined below). It could be I missed something since I am not familiar with Azuite but with the attached patch my use-case was working for me so I thought it was worth sharing. I have attached the patch here: Here is an (AI assisted) outline of the issues I ran into which the patch addresses: Issue 1: Fatal TypeError during DFS Append
Issue 2: Date Hydration Crash after Emulator Restart
Issue 3: Illegal Buffer.from() on Non-String Metadata
Summary of the Patch Set
Verification |
Wrap all lastModified values in new Date() before calling toUTCString()
so that string-typed values stored by LokiJS survive deserialization
without crashing. Coerce metadata values through String() before
base64-encoding to avoid Buffer type errors. Replace the fragile body
extraction in appendData with an explicit Buffer/string/array/object
branch so plain-object bodies (e.g. {}) are treated as empty rather
than causing a crash. Adds a regression test for both cases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
|
Patch applied |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 52 out of 56 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
src/blob/persistence/SqlBlobMetadataStore.ts:3618
conditionalPrefixReplaceExpr()builds aLIKE '${sourcePath}%'predicate usingsourcePathdirectly. IfsourcePathcontains%or_, the CASE expression can match and rewrite unrelated rows (wildcard expansion), even ifescapeLike()is fixed elsewhere. Ensure theLIKEpattern here escapes user-controlled wildcards consistently (and uses the same escaping strategy as theOp.likequeries).
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The stale first block described a single-blob rename with wrong param names; the second block is the accurate description. Drop the first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the sequential safeGetBlobProperties loop over returned prefixes with a single parallel Promise.all, eliminating the N+1 round-trip pattern that could significantly slow non-recursive directory listings at scale. Also applies the new Date() guard on lastModified for consistency with the rest of the handler. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/{fs}/{path}?resource=filesystem is not a valid DFS API URL shape.
Previously a GET on that pattern was silently routed to
Filesystem_ListPaths, producing a misleading 200. Collapse the two
ctx.path branches into one that always leaves operation undefined,
letting the dispatcher return 400 UnsupportedOperation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment on lines
+1207
to
+1217
| try { | ||
| await this.metadataStore.createBlob(createStorageContext(requestId), dirBlob); | ||
| // Register intermediate directory in HNS hierarchy | ||
| const parentDir = i > 1 ? parts.slice(0, i - 1).join("/") : null; | ||
| await this.metadataStore.registerHnsPath( | ||
| createStorageContext(requestId), account, filesystem, | ||
| dirPath, parentDir, true | ||
| ); | ||
| } catch { | ||
| // Ignore if already exists (race condition) | ||
| } |
Comment on lines
+3706
to
+3712
| await BlobsModel.update( | ||
| { | ||
| containerName: destContainer, | ||
| blobName: this.prefixReplaceExpr("blobName", sourcePrefix, destPrefix), | ||
| lastModified: now, | ||
| etag: newEtag() | ||
| } as any, |
Comment on lines
163
to
172
| containerHandler: new ContainerHandler( | ||
| this.accountDataStore, | ||
| this.oauth, | ||
| this.metadataStore, | ||
| this.extentStore, | ||
| this.logger, | ||
| this.loose | ||
| this.loose, | ||
| this.disableProductStyle, | ||
| this.enableHierarchicalNamespace | ||
| ), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds ADLS Gen 2 feature parity to Azurite. See docs/design/ADLS-gen2-parity.md file for details.
This was created by Claude Sonnet 4.6. Just sharing my progress here.
See OrchardCMS/OrchardCore#19014 for some integration tests.
The Github CI over there uses the current latest Docker image but I've succesfully ran my unit tests using a local Docker container based on this PR.
Maybe we need to create more unit tests. I let you guys decide, but it is looking good for a start.