Virtual File System for Node.js #61478
Conversation
|
Review requested:
|
nodejs-github-bot
added
lib / src
mcollina
requested review from
Ethan-Arrowood,
RaisinTen,
avivkeller and
joyeecheung
avivkeller
added
fs
|
The
notable-change
Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section. |
|
Nice! This is a great addition. Since it's such a large PR, this will take me some time to review. Will try to tackle it over the next week. |
| */ | ||
| existsSync(path) { | ||
| // Prepend prefix to path for VFS lookup | ||
| const fullPath = this.#prefix + (StringPrototypeStartsWith(path, '/') ? path : '/' + path); |
| /** | ||
| * Gets the underlying VirtualFileSystem instance. | ||
| * @returns {VirtualFileSystem} | ||
| */ | ||
| get vfs() { | ||
| return this.#vfs; | ||
| } | ||
|
|
||
| /** | ||
| * Gets the mount prefix for the mock file system. | ||
| * @returns {string} | ||
| */ | ||
| get prefix() { | ||
| return this.#prefix; | ||
| } |
There was a problem hiding this comment.
Do these need to be getters? Why can't we expose the actual values.
If a user overwrites them, they can if they wish?
| validateObject(files, 'options.files'); | ||
| } | ||
|
|
||
| const { VirtualFileSystem } = require('internal/vfs/virtual_fs'); |
There was a problem hiding this comment.
Shouldn't we import this at the top level / lazy load it at the top level?
| ArrayPrototypePush(this.#mocks, { | ||
| __proto__: null, | ||
| ctx, | ||
| restore: restoreFS, |
There was a problem hiding this comment.
| restore: restoreFS, | |
| restore: ctx.restore, |
nit
lib/internal/vfs/entries.js
Outdated
| * @param {object} [options] Optional configuration | ||
| */ | ||
| addFile(name, content, options) { | ||
| const path = this._directory.path + '/' + name; |
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const dirPath = '/' + segments.slice(0, i + 1).join('/'); |
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const parentPath = '/' + segments.slice(0, i + 1).join('/'); |
| } | ||
| } | ||
| callback(null, content); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
| }).catch((err) => { | |
| }, (err) => { |
lib/internal/vfs/virtual_fs.js
Outdated
| const bytesToRead = Math.min(length, available); | ||
| content.copy(buffer, offset, readPos, readPos + bytesToRead); |
| } | ||
|
|
||
| callback(null, bytesToRead, buffer); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
| }).catch((err) => { | |
| }, (err) => { |
|
Left an initial review, but like @Ethan-Arrowood said, it'll take time for a more in depth look |
|
It's nice to see some momentum in this area, though from a first glance it seems the design has largely overlooked the feedback from real world use cases collected 4 years ago: https://github.com/nodejs/single-executable/blob/main/docs/virtual-file-system-requirements.md - I think it's worth checking that the API satisfies the constraints that users of this feature have provided, to not waste the work that have been done by prior contributors to gather them, or having to reinvent it later (possibly in a breaking manner) to satisfy these requirements from real world use cases. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #61478 +/- ##
==========================================
+ Coverage 88.51% 88.93% +0.42%
==========================================
Files 704 676 -28
Lines 208883 207079 -1804
Branches 40334 39560 -774
==========================================
- Hits 184890 184163 -727
+ Misses 15977 15209 -768
+ Partials 8016 7707 -309
🚀 New features to boost your workflow:
|
|
And why not something like OPFS aka whatwg/fs? const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.mount('/app', rootHandle) // to make it work with fs
fs.readFileSync('/app/config.json')OR const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.readFileSync('sandbox:/config.json')fs.createVirtual seems like something like a competing specification |
mcollina
force-pushed
the
vfs
branch
2 times, most recently
from
f1cb562 to
5e317de
Compare
Add a read-only virtual file system (VFS) that can be mounted at a specific path prefix, enabling standard fs APIs to work transparently with in-memory files. Key features: - fs.createVirtual() to create VFS instances - Support for files, directories, and symbolic links - Full async/sync/promise API support (readFile, stat, readdir, etc.) - File descriptor operations (open, read, close) - createReadStream() support - fs.glob() integration - CJS require() and ESM import() support via module hooks - Virtual process.chdir() for relative path resolution - SEA integration via sea.getVfs() and sea.hasAssets() - Test runner mock.fs() for file system mocking The VFS is read-only by design and uses virtual file descriptors (10000+) to avoid conflicts with real file descriptors.
I generally prefer not to interleave with WHATWG specs as much as possible for core functionality (e.g., SEA). In my experience, they tend to perform poorly on our codebase and remove a few degrees of flexibility. (I also don't find much fun in working on them, and I'm way less interested in contributing to that.) On an implementation side, the core functionality of this feature will be identical (technically, it's missing writes that OPFS supports), as we would need to impact all our internal fs methods anyway. If this lands, we can certainly iterate on a WHATWG-compatible API for this, but I would not add this to this PR. |
|
Small prior art: https://github.com/juliangruber/subfs |
- Use path.isAbsolute() for cross-platform absolute path detection instead of checking for '/' prefix only - Use path.posix.join() in createScopedVFS to ensure forward slashes on all platforms since VFS uses '/' internally - Fix ESM module hook to recognize Windows absolute paths (C:\) - Fix symlink target resolution for Windows paths This enables VFS to work correctly on Windows where absolute paths use drive letters (e.g., C:\path) instead of Unix-style forward slashes.
Enable direct require() of modules from VFS in Single Executable
Applications. Previously, SEA's embedderRequire only supported built-in
modules, requiring users to use module.createRequire() for VFS paths.
Now, after calling sea.getVfs(), you can require VFS modules directly:
```js
const sea = require('node:sea');
sea.getVfs(); // Initialize VFS
const myModule = require('/sea/lib/mymodule.js');
```
The implementation:
- Lazily initializes SEA VFS on first non-builtin require
- Checks if the required path exists in VFS
- Uses Module._load to load from VFS via the registered hooks
7 participants
A first-class virtual file system that integrates with Node.js's fs module and module loader.
Key Features
fs.createVirtual()- Create isolated virtual file systems with files and directoriesrequire()and import work seamlessly from virtual files##Integrations
Example
Disclaimer: I've used a significant amount of Claude Code tokens to create this PR. I've reviewed all changes myself.