This is definitely a difficult problem given the relationship of Environment to the thread and the way the main thread is bootstrapped. The only way I could imagine being able to tackle this would be to redefine the main thread as "Just Another Type of Worker Thread" that is spun off a top level root thread.

That is, if we think about the current structure as a DAG, what we currently have is:

 (root thread 1) | ---> (worker thread 2)
  Environment    |         Environment
    Isolate      |           Isolate
                 |
                 | ---> (worker thread 3)
                           Environment
                             Isolate

To do what you're suggesting, we would likely need to move to a model like:

  (root thread) | ---> (worker thread 1) | ---> (worker thread 2)
                         Environment     |          Environment
                           Isolate       |            Isolate
                                         | ---> (worker thread 3)
                                                    Environment
                                                      Isolate

From there, we could allow a parent worker to be replaced by one of the children, but as you point out it gets complicated when we ask what happens with the other child.

  (root thread) | ---> (worker thread 1) | ---> (worker thread 2)  (replace parent with this thread)
                         Environment     |         Environment
                           Isolate       |           Isolate
                                         | ---> (worker thread 3)
                                                   Environment
                                                     Isolate

  (root thread) | ---> (worker thread 2)
                |         Environment
                |           Isolate
                | ---> (worker thread 3)
                          Environment
                            Isolate

There are a couple of options. (a) We could terminate orphan workers automatically, (b) We could just leave them orphaned, (c) We could transfer responsibility for them to the new parent.

I think we can all agree that (b) is not a great choice. We can let the user code decide between (a) and (c).

For (c), the DAG would end up as:

  (root thread) | ---> (worker thread 2)  | ---> (worker thread 3)
                          Environment               Environment
                            Isolate                   Isolate

Obviously there are a ton of complexities wrapped up in that!

The key question I would have initially is: Is replacing threads really what we need? Alternatively, we could implement the ability for a Worker Thread to transfer ownership/responsibility for a child thread to a different thread.

For instance, the Node.js main process thread can load the apm implementation in a separate thread, which does its thing and spawns the instrumented application code thread. Once the application code thread is spun up, the APM thread transfers ownership of that back to the main thread then exits. This, however, obviously does not provide the requirement of the spun up worker acting like it's the main thread (no piped i/o, etc)

The other possibility is whether APM's could just be written to act as alternative launchers, or embedders, of the Node.js process so that they can inject the loaders the extensive code changes that would make replacing the thread possible. I know that has its own associated complexities, however.

The key question I would have initially is: Is replacing threads really what we need? Alternatively, we could implement the ability for a Worker Thread to transfer ownership/responsibility for a child thread to a different thread.

Well, the problem is only the main thread can act like a main thread given how worker_threads are implemented, we could also make this just be some form of allowing worker threads to look and act exactly like the main thread? I didn't want to approach it that way since things can break if running in worker threads and they don't have the proper APIs due to limitations on worker_threads.

For instance, the Node.js main process thread can load the apm implementation in a separate thread, which does its thing and spawns the instrumented application code thread. Once the application code thread is spun up, the APM thread transfers ownership of that back to the main thread then exits.

This would seem fine to me, but still has the problem of the application living in a worker_thread environment which doesn't exactly match the main thread environment. So, this doesn't seem viable and more than other approaches.

The other possibility is whether APM's could just be written to act as alternative launchers, or embedders, of the Node.js process so that they can inject the loaders the extensive code changes that would make replacing the thread possible. I know that has its own associated complexities, however.

How would this be any more able to solve the real world issues than the existing --loader where they do not completely control the env they are running on? Also, doing this as custom embedders seems even harder to compose multiple APMs than --loader.

None given how ESM works and links eagerly this is the only solution I've found.

Theoretically couldn't you use a cjs entrypoint that calls import()?

@devsnek if the loader hasn't been used yet maybe (--require can call out to import)? but it also wouldn't solve for ESM entrypoints.

How would this be any more able to solve the real world issues than the existing --loader where they do not completely control the env they are running on? Also, doing this as custom embedders seems even harder to compose multiple APMs than --loader.

As I said, it has it's own complexities ;-) ... there's not going to be an easy and simple solution to the problem, unfortunately. At this point I'm just saying we need to consider alternatives and rule them out to try to narrow in on the least complicated and difficult approach. I'm nowhere near convinced that the work necessary to make threads replaceable is it.

As I said, it has it's own complexities ;-) ... there's not going to be an easy and simple solution to the problem, unfortunately. At this point I'm just saying we need to consider alternatives and rule them out to try to narrow in on the least complicated and difficult approach.

I'd note that solving for "least complicated and difficult" varies wildly on what you are trying to optimize for. I think we should optimize for some usability beyond custom builds of node since this space is a fairly generic one.

I'm nowhere near convinced that the work necessary to make threads replaceable is it.

As I've stated above, allowing worker threads to act exactly like the main thread would be sufficient as well. Though it would mean effectively leave a dead thread if the original isn't destroyed I'd assume. Also, I would not I am not talking about "native" threads here but conceptual ones. I could easily imagine the replace behavior spinning up an Isolate in the same native thread.

I could easily imagine the replace behavior spinning up an Isolate in the same native thread.

This is an interesting approach also, and one that I've been stewing over... essentially, allowing multiple Environment/Isolates to exist within the same thread and sharing the same event loop. I was hoping to get around to playing with this soon.