(Windows) FS can not handle certain characters in file name

Version

v20.4.0

Platform

Microsoft Windows NT 10.0.19045.0 x64

Subsystem

No response

What steps will reproduce the bug?

Run the following script

require('fs').writeFileSync('\uD83D\uDD79\uFE0F.log','Test')

Where \uD83D\uDD79\uFE0F is 🕹️

How often does it reproduce? Is there a required condition?

Always failed for v20.4.0, v20.3.1 is okay

What is the expected behavior? Why is that the expected behavior?

Run without errors

What do you see instead?

Additional information

Tried on linux, no errors

@vtjnash

Libuv recently switched to WTF-8, to deal with surrogate pairs (0xD83D 0xDD79 is a surrogate pair) in Windows file paths. That change landed in v20.4.0.

cc @vtjnash

\uD83D\uDD79\uFE0F is not a legal string, since it incorrectly encodes for a surrogate, not a unicode character. The Joystick emoji encoding is:

U+1F579 ️ U+FE0F

We could potentially disable the UTF-8 transcoder error checking again though, and permit this to access the file anyways. The name would not round-trip correctly through the file-system, but that is not too significantly different from how case-sensitivity can be lost during the conversion to accessing the file system.

\uD83D\uDD79\uFE0F is not a legal string, since it incorrectly encodes for a surrogate, not a unicode character. The Joystick emoji encoding is:

U+1F579 ️ U+FE0F

For what reason you think it is not valid?

'\u{1F579}'==='\uD83D\uDD79' // true

This is JavaScript, they are identical.

Ah, interesting. That means node is explicitly converting the valid string in ecmascript's UCS-2 into an invalid UTF-8 string, possibly intentionally since ecmascript does not actually support this emoji and instead defines that it is composed of 2 unknown characters from the surrogate block. We could certainly change libuv to be a non-validating UTF-8 decoder so that it can still accept invalid strings as input.

https://mathiasbynens.be/notes/javascript-encoding

fs: fix WTF-8 decoding issue

fs: fix WTF-8 decoding issue (#4092)

This maybe shouldn't be closed until we get a libuv release then a Node release that includes it? We're going to collect a lot of dupes for this and they should be closed and point here until the release is done. #48813 #49042.

(Speaking from experience, I'm dealing with some failing windows CI tests since 20.4.0 https://github.com/ipld/codec-fixtures/actions/runs/5756528910/job/15606085523 for files that have ... complicated ... names).

LTS is also affected. Works in v18.0.0, but not in v18.18.0.

It's fixed upstream in libuv/libuv@d09441c but not released yet.

Disables emoji in our Windows tests until Node is fixed (#5763)

[95d8a273cc] - deps: cherry-pick bfbe4e38d7 from libuv upstream (Abdirahim Musse) deps: update libuv to 1.47.0 #50650
[06038a489e] - deps: update libuv to 1.47.0 (Node.js GitHub Bot) deps: update libuv to 1.47.0 #50650

were added only in "Current" (21.x) version, not in "LTS" (20.x) now.

@AlttiRi

As @AlttiRi points out, I think it's shouldn't be closed until it's included in LTS.

@rvagg What do you think?

yeah, I agree, I'm still getting LTS failures but am sick of waiting so am about to add an exclusion to my CI rules just for this case so I'll leave it to others to litigate

Thanks for adding the fix to current version, could someone follow up including the fix in LTS version? Thanks!

We're using 20.3.1 version, could we had the fix backported into this version as well?

~~It was finally fixed in LTS(5405aa5).~~

~~[v20.x backport] deps: upgrade libuv to 1.48.0 #51699~~

WTF?

import fs from "node:fs/promises";
await fs.writeFile("🚀🔥🛸.txt", "");

node:internal/fs/promises:609
    await binding.openFileHandle(pathModule.toNamespacedPath(path),
                  ^

Error: ENOENT: no such file or directory, open
    at open (node:internal/fs/promises:609:19)
    at Object.writeFile (node:internal/fs/promises:1055:20)
    at file:///C:/Dev/test/x.mjs:2:10
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'open'
}

Node.js v20.11.1

Also, I've seen these errors:

node:internal/fs/utils:557
    stats[0 + offset], stats[1 + offset], stats[2 + offset],
         ^

TypeError: Cannot read properties of undefined (reading '0')
    at getStatsFromBinding (node:internal/fs/utils:557:10)
    at Object.lstat (node:internal/fs/promises:917:10)
    at async file:///C:/Dev/test/x.mjs:3:1

node:internal/fs/promises:915
  const result = await binding.lstat(pathModule.toNamespacedPath(path),
                               ^

Error: ENOENT: no such file or directory, lstat
    at Object.lstat (node:internal/fs/promises:915:32)
    at file:///C:/Downloads/1/x.mjs:3:16
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:28:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'lstat'
}

Node.js v20.11.1

It's still not fixed. Even with libuv 1.48.

Even Bun https://github.com/oven-sh/bun/ works fine with it.

UPD.

However, it's fixed in Current (v21.6.2).

But it's not fixed in LTS (v20.11.1) that was released 3 day ago.

> LTS (Long-Term Support). Only verified changes.
> "Recommended For Most Users"
> broken

Are you kidding?

Node.js even have not tests for reading/writing files with surrogate pairs in a filename?

Do we have a release date for this bug fix?

I assume, in the next LTS release after merging that PR:

[v20.x] backport libuv wtf-8 decoding fix #51976

It takes more than half a year to add 3 lines of code.

deps: cherry-pick libuv/libuv@d09441c

test: test surrogate pair filenames on windows

Finally, the fix was backported to LTS (v20.12.0).

v20.12.0 proposal #52212:
- [cb49f31480] - deps: cherry-pick libuv/libuv@d09441c (Richard Lau) [v20.x] backport libuv wtf-8 decoding fix #51976
- [e8f5735149] - test: test surrogate pair filenames on windows (Mert Can Altın) test: Test surrogate pair filenames on windows #51800
Rel: Add tests for filenames with surrogate pairs on Windows #51789

2023-10-10, Version 18.18.1 'Hydrogen' (LTS)

Navigation Menu

Repository navigation

(Windows) FS can not handle certain characters in file name #48673

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

(Windows) FS can not handle certain characters in file name #48673

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

bnoordhuis commented on Jul 6, 2023

vtjnash commented on Jul 6, 2023

vtjnash commented on Jul 6, 2023

ZEDCWT commented on Jul 7, 2023

vtjnash commented on Jul 7, 2023

rvagg commented on Aug 7, 2023

user7230724 commented on Sep 21, 2023

bnoordhuis commented on Sep 26, 2023

12 remaining items

AlttiRi commented on Dec 21, 2023

kzrnm commented on Dec 29, 2023

rvagg commented on Dec 29, 2023

qianz2 commented on Jan 8, 2024

qianz2 commented on Jan 10, 2024

AlttiRi commented on Feb 17, 2024

Rinse12 commented on Mar 14, 2024

AlttiRi commented on Mar 14, 2024

AlttiRi commented on Mar 30, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions