Bug Report

Search Terms

• EMFILE: too many open files, watch
• fsevents
• FSEventStreamStart

馃晽 Version & Regression Information

TypeScript version: 4.5.4
macOS version: 11.6.2
node version: v17.4.0

馃捇 Code

This issue is macOS specific.

git clone git@github.com:gluxon/typescript-emfile-repro
npm install
npm run repro

The repository above contains 2050 project references. Each of these projects have only an empty index.ts file to keep the repro simple.

The solution style tsconfig at the root is set to use fsevents.

{
  "watchOptions": {
    "watchFile": "useFsEventsOnParentDirectory",
    "watchDirectory": "useFsEvents"
  },
  "references": [
    // ...
  ]
}

The 2050 project references may seem unreasonable, but in a large monorepo this becomes more realistic when factoring in node_modules and nested watchers. (See below.) The real world repository this issue appeared on has around 2 million lines of TypeScript and ~800 packages declaring project references on each other.

馃檨 Actual behavior

When running the repro linked above, tsc eventually switches all existing watchers to a MissingFileSystemEntryWatcher.

[11:48:15 AM] Found 0 errors. Watching for file changes.

DirectoryWatcher:: Added:: WatchInfo: /Volumes/git/typescript-emfile-repro 1 {"watchFile":5,"watchDirectory":0} Wild card directory
Elapsed:: 0.4360110005363822ms DirectoryWatcher:: Added:: WatchInfo: /Volumes/git/typescript-emfile-repro 1 {"watchFile":5,"watchDirectory":0} Wild card directory
sysLog:: /Volumes/git/typescript-emfile-repro:: Changing watcher to MissingFileSystemEntryWatcher
sysLog:: /Volumes/git/typescript-emfile-repro/packages/0:: Changing watcher to MissingFileSystemEntryWatcher
sysLog:: /Volumes/git/typescript-emfile-repro/packages/0:: Changing watcher to MissingFileSystemEntryWatcher

This behavior comes from the watchPresentFileSystemEntry function here.

Updating the presentWatcher.on("error") listener to print the error shows:

presentWatcher.on("error", (err) => {
  console.error(err);
  invokeCallbackAndUpdateWatcher(watchMissingFileSystemEntry);
});

Error: EMFILE: too many open files, watch
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:168:28)
Emitted 'error' event on FSWatcher instance at:
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:174:12) {
  errno: -24,
  syscall: 'watch',
  code: 'EMFILE',
  filename: null
}

Once EMFILE errors start emitting here, TypeScript no longer responds to any .ts file edits in in the repository since watchMissingFileSystemEntry only responds to FileWatcherEventKind.Created, and these files already exist.

From testing on my MacBook Pro (15-inch, 2018) on macOS 11.6.2, the number of fs.watch() calls that triggers EMFILE seems to be around ~4099.

馃檪 Expected behavior

I think it makes sense to update the error event listener to no longer assume all fs.FSWatcher errors are due to a missing file/directory, but I'm actually not sure what TypeScript should in response to a EMFILE after that. Ideally TypeScript falls back to using the polling watcher like it does for Linux.

However in my tests, once a Node.js program hits EMFILE on an fs.FSWatcher instance, all FSWatcher instances start throwing on EMFILE too. Because of the Node.js APIs here, I think TypeScript has to revert fully to polling mechanisms.

Hoping for ideas from the TypeScript team. Thanks in advance. 馃檪

Additional Research

Notes from libuv

Adding some additional research into Node.js and libuv.

I was surprised that ulimit -n unlimited had no effect on the repro. It turns out the EMFILE error here is from libuv, not a syscall.

https://github.com/libuv/libuv/blob/47e0c5c575e92a25e0da10fc25b2732942c929f3/src/unix/fsevents.c#L380-L384

if (!pFSEventStreamStart(ref)) {
  pFSEventStreamInvalidate(ref);
  pFSEventStreamRelease(ref);
  return UV_EMFILE;
}

In Node.js land that error gets emitted here:

https://github.com/nodejs/node/blob/ec1364b6ae8f3be9054af5c56c7bf7c2d1b7c735/lib/internal/fs/watchers.js#L204-L210

const error = uvException({
  errno: status,
  syscall: 'watch',
  path: filename
});
error.filename = filename;
this.emit('error', error);

Looking at FSEventStreamStart documentation from Apple:

Return Value

True if it succeeds, otherwise False if it fails. It ought to always succeed, but in the event it does not then your code should fall back to performing recursive scans of the directories of interest as appropriate.

fs.watch Logging

Tangential to this issue, it may be possible for TypeScript to optimize its fs.watch calls. In the case below, the src is set to be recursively watched, but TypeScript ends up adding a few more watchers under it.

fs.watch("/Volumes/example/packages/foo", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src", { persistent: true, recursive: true });
fs.watch("/Volumes/example/packages/foo/typings", { persistent: true, recursive: true });
fs.watch("/Volumes/example/packages/foo/src", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src/__tests__", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src/bar", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/src/bar/__tests__", { persistent: true, recursive: false });
fs.watch("/Volumes/example/packages/foo/typings", { persistent: true, recursive: false });

From debugging, it does appear to be the number of fs.watch calls made that triggers the EMFILE error, rather than the number of files in a directory or its storage size. For example, watching the root of a large monorepo package does not cause EMFILE in the repro:

fs.watch("/Volumes/git/typescript-emfile-repro", { persistent: true, recursive: true })