Worker threads: ability for tooling to pass data to workers and inject NODE_OPTIONS #30992

coreyfarrell · 2019-12-16T18:29:12Z

Is your feature request related to a problem? Please describe.
nyc 15 is moving away from spawn-wrap, will use process-on-spawn and node-preload in it's place. These two modules ensure nyc gets properly loaded into new processes, but they do not help with worker threads (note spawn-wrap also did nothing to help with worker threads). I'd like to add worker thread support to nyc.

Describe the solution you'd like
nyc needs to pass data to worker threads without requiring modifications to arguments of new Worker(...) run by the code being tested. An idea I have:

process.on('worker', options => {
	options.platformData('nyc', nycConfigClonableObject);
	options.nodeOptions.push('--require', require.resolve('./lib/wrap-thread.js'));
});

The above code assumes that new Worker() internally calculates options.nodeOptions based on the env.NODE_OPTIONS passed to it (see also #29117). This would allow nyc to inject preload modules if necessary without altering the process.env of the new thread.

Describe alternatives you've considered
Setting platform data during process startup could work for some use cases:

const threads = require('worker_threads');
threads.platformData('nyc', dataNeededToInitializeNYC);
threads.platformNodeOptions.push('--require', require.resolve('./wrap-thread.js')));

This would work for nyc but maybe another tool would need to calculate the object needed for platformData at the time of new Worker(). This is why my proposal is for 'worker spawn' hooks to ensure this is useful for more than just nyc. I'm also unsure how platformNodeOptions would be incorporated into the env.NODE_OPTIONS given to new Worker().

I considered patching the worker_threads core module. worker.SHARE_ENV is very problematic for this approach as it eliminates the possibility of using environmental variables to get nyc options into the worker threads. We would likely need to patch new Worker() to wrap worker data so that options.workerData = {userData: options.workerData, nycData}, then require('worker_threads').workerData would be patched to return the userData subproperty and we'd create a require('worker_threads').nycData export. This seems like it would be very messy, especially if any other tooling had a need to patch workerData in a similar way. I'm really not interested in this approach even if it could be forced to work.

CC @addaleax @arcanis @nodejs/tooling

The text was updated successfully, but these errors were encountered:

bmeck · 2020-01-06T15:48:32Z

It seems like the event listener approach would at least allow some censoring/protecting of options? You could use Object.freeze etc on the options?

coreyfarrell · 2020-01-07T19:56:59Z

@bmeck I prefer the event listener approach since it allows calculation of data and NODE_OPTIONS to happen when the thread is being created instead of needing to know those values way ahead of time, I'm not sure it helps give any special protection to the options that are specified. One drawback of using the events API is that process.removeAllListeners('worker') could remove my hook. An alternative to the event listener approach would be an API similar to async_hooks where the worker hook would only be removable by the code which added it.

I'm not sure options is actually the best name for the argument either, I'm very open to suggestions for how to best structure the API. Maybe context would be a better name since I was really thinking this would be a class instance with methods to manage data and node options.

Given the PR to run loader hooks in a worker thread I have to ask how you see this interacting with loader hooks? I assume loader hooks thread will always start before any --require's are preloaded and those preloads would not be run inside the loader worker thread? Sorry if this is answered in the PR I really haven't had a chance to dig into the code yet.

bmeck · 2020-01-07T20:13:02Z

I assume loader hooks thread will always start before any --require's are preloaded and those preloads would not be run inside the loader worker thread? Sorry if this is answered in the PR I really haven't had a chance to dig into the code yet.

That is my presumption yes.

The `worker.platformData` and `worker.setPlatformData()` APIs allow an arbitrary, cloneable JavaScript value to be set and passed to all new Worker instances spawned from the current context. It is similar to `workerData` except that `platformData` is set independently of the `new Worker()` constructor, and the the value is passed automatically to all new Workers. This is a *partial* fix of nodejs#30992 but does not implement a complete fix. Signed-off-by: James M Snell <jasnell@gmail.com>

These APIs allow arbitrary, cloneable JavaScript values to be set and passed to all new Worker instances spawned from the current context. It is similar to `workerData` except that environment data is set independently of the `new Worker()` constructor, and the the value is passed automatically to all new Workers. This is a *partial* fix of nodejs#30992 but does not implement a complete fix. Signed-off-by: James M Snell <jasnell@gmail.com>

These APIs allow arbitrary, cloneable JavaScript values to be set and passed to all new Worker instances spawned from the current context. It is similar to `workerData` except that environment data is set independently of the `new Worker()` constructor, and the the value is passed automatically to all new Workers. This is a *partial* fix of #30992 but does not implement a complete fix. Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37486 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Darshan Sen <raisinten@gmail.com>

devsnek added worker feature request labels Dec 20, 2019

jasnell mentioned this issue Feb 23, 2021

worker: add support for platformData #37486

Closed

nodejs / node

Worker threads: ability for tooling to pass data to workers and inject NODE_OPTIONS #30992

Worker threads: ability for tooling to pass data to workers and inject NODE_OPTIONS #30992

coreyfarrell commented Dec 16, 2019

bmeck commented Jan 6, 2020

coreyfarrell commented Jan 7, 2020

bmeck commented Jan 7, 2020

Feb	MAR	Apr
	18
2020	2021	2022

nodejs / node

Worker threads: ability for tooling to pass data to workers and inject NODE_OPTIONS #30992

Worker threads: ability for tooling to pass data to workers and inject NODE_OPTIONS #30992

Comments

coreyfarrell commented Dec 16, 2019

bmeck commented Jan 6, 2020

coreyfarrell commented Jan 7, 2020

bmeck commented Jan 7, 2020