src: add initial support for single executable applications #45038

RaisinTen · 2022-10-17T13:47:10Z

Compile a JavaScript file into a single executable application:

$ cat hello.js
console.log(`Hello, ${process.argv[2]}!`);
$ cp $(command -v node) hello
$ npx postject hello NODE_JS_CODE hello.js
$ ./hello world
Hello, world!

Signed-off-by: Darshan Sen raisinten@gmail.com

It was decided in the Node.js Collaborator Summit that we want to come up with an MVP that is capable of compiling a single JavaScript file into a single executable application. More features will be added incrementally.

Meeting notes for reference: https://github.com/nodejs/single-executable/blob/c7008ecd67a395c7c902d602f2f116e2c5744bc9/meetings/2022-10-01.md

Note for reviewers: The change inside deps/ has been moved to #45066, so if you have any concerns regarding that portion of this PR, feel free to post your comments in #45066!

cc @nodejs/single-executable

nodejs-github-bot · 2022-10-17T13:47:14Z

Review requested:

Refs: nodejs/node#45038 Signed-off-by: Darshan Sen <raisinten@gmail.com>

doc/api/cli.md

lib/internal/main/compile.js

lib/internal/main/single_executable_application.js

tools/update-postject.sh

.github/workflows/tools.yml

jviotti · 2022-10-17T15:33:23Z

@RaisinTen Don't forget to list this PR in https://github.com/nodejs/single-executable's README, for the record :)

kapv89 · 2022-10-17T16:03:18Z

Question: does the executable require node to be installed?

deps/postject/src/package.json

jasnell · 2022-10-17T17:51:15Z

Adding postject really should be done in a separate commit. Also, are we sure it's actually needed here at all? It's a rather sizeable addition on its own.

GeoffreyBooth · 2022-10-17T18:34:50Z

Please make it print the experimental warning if that isn’t happening already.

Fixes the following error I came across while integrating Postject in Node.js: ```console ../deps/postject/src/dist/postject-api.h:30:13: error: unused function 'postject_options_init' [-Werror,-Wunused-function] static void postject_options_init(struct postject_options* options) { ^ 1 error generated. ``` Refs: nodejs/node#45038 Signed-off-by: Darshan Sen <raisinten@gmail.com>

ovflowd · 2022-10-19T08:51:40Z

Is it just me or the deps/postject/postject.js is obfuscated, or at least very unreadable? Is this intended? Is this the source of the project or a compiled version? (I believe the later one)

ovflowd · 2022-10-19T08:54:32Z

Off-topic: Having the source of postject here means that this is a full copy of the original repository? What would happen with the original repository? Or is this intended to be a snapshot of the project at a given point in time?

ovflowd

Super cool stuff! Super mega hyper excited for this!

deps/postject/src/README.markdown

doc/api/cli.md

src/node.cc

src/util.cc

test/parallel/test-single-executable-application.js

tools/update-postject.sh

This change has been extracted from the single executable application PR - nodejs#45038. I'm doing this in a separate PR because as James said in nodejs#45038 (comment), it is a rather sizeable addition on its own, so it should be easier to review and address related changes in this PR. This dependency takes care of the resource injection part of the single executable application work. Refs: https://github.com/nodejs/single-executable/blob/34b6765a871222e142f9dbac2bcde5b4e79ebfb2/blog/2022-08-05-an-overview-of-the-current-state.md#resource-injection Signed-off-by: Darshan Sen <raisinten@gmail.com>

RaisinTen · 2022-10-19T09:40:08Z

Question: does the executable require node to be installed?

@kapv89 no it does not. That's one of the main use cases for the single executables applications - so that it is possible to distribute a Node.js application to a user who does not already have Node.js installed on their system.

This change has been extracted from the single executable application PR - nodejs#45038. I'm doing this in a separate PR because as James said in nodejs#45038 (comment), it is a rather sizeable addition on its own, so it should be easier to review and address related changes in this PR. This dependency takes care of the resource injection part of the single executable application work. Refs: https://github.com/nodejs/single-executable/blob/34b6765a871222e142f9dbac2bcde5b4e79ebfb2/blog/2022-08-05-an-overview-of-the-current-state.md#resource-injection Signed-off-by: Darshan Sen <raisinten@gmail.com>

RaisinTen · 2022-10-19T09:50:04Z

Adding postject really should be done in a separate commit. Also, are we sure it's actually needed here at all? It's a rather sizeable addition on its own.

@jasnell sure, moved it to its own PR - #45066. And yes, it is needed for the resource injection part.

RaisinTen · 2022-10-19T09:52:54Z

Is it just me or the deps/postject/postject.js is obfuscated, or at least very unreadable? Is this intended? Is this the source of the project or a compiled version? (I believe the later one)

@ovflowd it is the compiled version :)

ovflowd · 2022-10-19T10:57:38Z

@ovflowd it is the compiled version :)

Do we need the compiled version? As node-gyp seems to actually build the source?

RaisinTen · 2022-10-19T11:13:32Z

@ovflowd as mentioned in #45066 (comment), that is the webassembly code which is precompiled in postject. It's built using the emsdk, not node-gyp.

* fix: make sure that postject-api.h compiles without warnings Fixes the following error I came across while integrating Postject in Node.js: ```console ../deps/postject/src/dist/postject-api.h:30:13: error: unused function 'postject_options_init' [-Werror,-Wunused-function] static void postject_options_init(struct postject_options* options) { ^ 1 error generated. ``` Refs: nodejs/node#45038 Signed-off-by: Darshan Sen <raisinten@gmail.com> * fix: resolve another compiler warning ```console In file included from /root/project/test/test.c:4: /root/project/test/../dist/postject-api.h: In function 'postject_find_resource': /root/project/test/../dist/postject-api.h:96:9: error: unused variable 'ptr' [-Werror=unused-variable] 96 | void* ptr = NULL; | ^~~ cc1: all warnings being treated as errors ``` Refs: https://app.circleci.com/pipelines/github/postmanlabs/postject/180/workflows/d0eb47c0-5482-4c85-9c63-aa854ddb0221/jobs/1398?invite=true#step-110-678 Signed-off-by: Darshan Sen <raisinten@gmail.com> * fix: use -Wall -WX on Windows Signed-off-by: Darshan Sen <raisinten@gmail.com> * chore: use -W4 instead of -Wall -Wall produces too many warnings. Refs: https://app.circleci.com/pipelines/github/postmanlabs/postject/185/workflows/48b0f126-8000-41d9-b39e-cb8b9e4bc9d6/jobs/1441?invite=true#step-107-693 Signed-off-by: Darshan Sen <raisinten@gmail.com> * fix: another compilation warning on Windows ```console C:\Users\circleci\project\test\../dist/postject-api.h(153,5): error C2220: the following warning is treated as an error [C:\Users\circleci\project\build\test\c_test.vcxproj] C:\Users\circleci\project\test\../dist/postject-api.h(153,5): warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. [C:\Users\circleci\project\build\test\c_test.vcxproj] C:\Users\circleci\project\test\test.c(13,5): warning C4996: 'strncpy': This function or variable may be unsafe. Consider using strncpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. [C:\Users\circleci\project\build\test\c_test.vcxproj] ``` Refs: https://app.circleci.com/pipelines/github/postmanlabs/postject/186/workflows/f1389b9f-c958-4a24-9d6e-28af856ff776/jobs/1455?invite=true#step-107-694 Signed-off-by: Darshan Sen <raisinten@gmail.com> * fix: compiler warning on Windows ```console C:\Users\circleci\project\test\test.c(13,5): error C2220: the following warning is treated as an error [C:\Users\circleci\project\build\test\c_test.vcxproj] C:\Users\circleci\project\test\test.c(13,5): warning C4996: 'strncpy': This function or variable may be unsafe. Consider using strncpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details. [C:\Users\circleci\project\build\test\c_test.vcxproj] ``` Refs: https://app.circleci.com/pipelines/github/postmanlabs/postject/187/workflows/8bcdcb96-646d-4008-93bf-294e92469b3d/jobs/1464?invite=true#step-107-694 Signed-off-by: Darshan Sen <raisinten@gmail.com> * fix: use -EHsc for test.cpp ```console C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\ostream(743,1): error C2220: the following warning is treated as an error [C:\Users\circleci\project\build\test\cpp_test.vcxproj] C:\Users\circleci\project\test\test.cpp(13): message : see reference to function template instantiation 'std::basic_ostream<char,std::char_traits<char>> &std::operator <<<std::char_traits<char>>(std::basic_ostream<char,std::char_traits<char>> &,const char *)' being compiled [C:\Users\circleci\project\build\test\cpp_test.vcxproj] C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\include\ostream(743,1): warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc [C:\Users\circleci\project\build\test\cpp_test.vcxproj] ``` Refs: https://app.circleci.com/pipelines/github/postmanlabs/postject/188/workflows/9107adc5-61a3-41ad-bd60-dd3eb0996765/jobs/1470?invite=true#step-107-696 Signed-off-by: Darshan Sen <raisinten@gmail.com> * fix: use / instead of - for Windows compiler options Signed-off-by: Darshan Sen <raisinten@gmail.com> * chore: use target_compile_options instead of set Signed-off-by: Darshan Sen <raisinten@gmail.com> Signed-off-by: Darshan Sen <raisinten@gmail.com>

mhdawson · 2022-10-20T21:30:42Z

@RaisinTen now that there is a separate PR to discuss how/where postject would be included, does it makes sense to remove that part from this PR so that it's easier to see just the key parts that are needed outside of postject?

arcanis · 2022-10-21T13:59:31Z

And yes, it is needed for the resource injection part.

I don't understand why it needs to be in the Node binary, to be honest (or even distributed with the main Node distribution).

It seems to me only a very, very small handful of people would use it, and only when publishing applications - so why isn't it something left to userland to install, through a regular dependency (like is node-gyp, for example, which is massively more used than postject would be)?

RaisinTen · 2022-10-21T14:11:13Z

@RaisinTen now that there is a separate PR to discuss how/where postject would be included, does it makes sense to remove that part from this PR so that it's easier to see just the key parts that are needed outside of postject?

@mhdawson I have removed that part, PTAL.

I don't understand why it needs to be in the Node binary

@arcanis you can take a look at the second list of points I mentioned in #45066 (comment).

mhdawson · 2022-10-21T17:21:47Z

doc/api/single-executable-applications.md

+This feature allows the distribution of a Node.js application conveniently to a
+system that does not have Node.js installed.
+
+A bundled JavaScript file can be turned into a single executable application


We should probably also add a comment that it can also be done with any other tool which can inject sections.

How can we use any other tools? Can you add them to the instructions?

objcopy can be used to accomplish this. I've created an issue in postject to discuss if it makes sense to use it internally: nodejs/postject#63.

However, after nodejs/postject#59 lands, it would also add another step - flipping the bits in the fuse. This can't be done purely by a section injection tool.

mhdawson · 2022-10-21T17:29:16Z

src/util.cc

@@ -475,4 +480,19 @@ void SetConstructorFunction(Local<v8::Context> context,
  that->Set(context, name, tmpl->GetFunction(context).ToLocalChecked()).Check();
 }

+const char* FindSingleExecutableCode(size_t* size) {
+  if (single_executable_application_code_loaded == false) {


I think we need the concept of a "fuse" here. Something like what I had in src/node_single_executable_application.cc on line 43, as borrowed from electron.

That ensures that if you are not using SEA, the overhead is as close to possible as 0. It does mean there is an extra step in the bundling but I don't see that as a big issue.

@mhdawson does nodejs/postject#59 look good to you?

mhdawson · 2022-10-21T17:34:22Z

doc/api/single-executable-applications.md

@@ -0,0 +1,44 @@
+# Single executable applications


In addition to this file I think we need something like doc/contributing/maintaining-single-executable-application-support.md in this PR which explains the strategy, layout of what's injected etc.

deps/postject/src/dist/postject-api.h

No need for it. Refs: nodejs/node#45038 (comment) Signed-off-by: Darshan Sen <raisinten@gmail.com>

chore: remove Postman copyright from postject-api.h No need for it. Refs: nodejs/node#45038 (comment) Signed-off-by: Darshan Sen <raisinten@gmail.com> Signed-off-by: Darshan Sen <raisinten@gmail.com>

Compile a JavaScript file into a single executable application: ```console $ cat hello.js console.log(`Hello, ${process.argv[2]}!`); $ cp $(command -v node) hello $ npx postject hello NODE_JS_CODE hello.js $ ./hello world Hello, world! ``` Signed-off-by: Darshan Sen <raisinten@gmail.com>

RaisinTen · 2022-10-31T11:01:21Z

test/internet/test-single-executable-application.js

@@ -0,0 +1,79 @@
+'use strict';


Does anyone know how I can run this test on CI? I have placed it inside test/internet instead of test/parallel because it attempts to do npx postject which makes an outbound network connection.

I'm thinking VCR_Cassettes, like in Ruby?

On Node's CI runners? Or any sort of CI?

During the single-executable meeting, we were discussing about including postject inside test/fixtures, just for testing. So that might solve this problem. PR to do that - #45298.

Refs: nodejs#45038 (comment) Signed-off-by: Darshan Sen <raisinten@gmail.com>

This should make it possible to test out the creation of Single Executable Applications on a PR without making outbound requests to download and run postject using npm. This is a needed for nodejs#45038. Refs: https://github.com/nodejs/single-executable/blob/1840f3d9c5f4fa0d29aabd5618c4ff9745f7be87/meetings/2022-10-31.md?plain=1#L75-L76 Signed-off-by: Darshan Sen <raisinten@gmail.com>

This should make it possible to test out the creation of Single Executable Applications on a PR without making outbound requests to download and run postject using npm. This is needed for nodejs#45038. Refs: https://github.com/nodejs/single-executable/blob/1840f3d9c5f4fa0d29aabd5618c4ff9745f7be87/meetings/2022-10-31.md?plain=1#L75-L76 Signed-off-by: Darshan Sen <raisinten@gmail.com>

This should make it possible to test out the creation of Single Executable Applications on a PR without making outbound requests to download and run postject using npm. This is needed for #45038. Refs: https://github.com/nodejs/single-executable/blob/1840f3d9c5f4fa0d29aabd5618c4ff9745f7be87/meetings/2022-10-31.md?plain=1#L75-L76 Signed-off-by: Darshan Sen <raisinten@gmail.com> PR-URL: #45298 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Tierney Cyren <hello@bnb.im>

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Oct 17, 2022

RaisinTen mentioned this pull request Oct 17, 2022

src,doc: Experimental support for SEA #42334

Draft

RaisinTen added a commit to nodejs/single-executable that referenced this pull request Oct 17, 2022

Add a link to the initial MVP to the status

5903bf8

Refs: nodejs/node#45038 Signed-off-by: Darshan Sen <raisinten@gmail.com>

RaisinTen mentioned this pull request Oct 17, 2022

Add a link to the initial MVP to the status nodejs/single-executable#52

Merged

aduh95 added semver-minor PRs that contain new features and should be released in the next minor version. notable-change PRs with changes that should be highlighted in changelogs. labels Oct 17, 2022

aduh95 reviewed Oct 17, 2022

View changes

.github/workflows/tools.yml Outdated Show resolved Hide resolved

ruyadorno reviewed Oct 17, 2022

View changes

deps/postject/src/package.json Outdated Show resolved Hide resolved

RaisinTen mentioned this pull request Oct 19, 2022

fix: make sure that postject-api.h compiles without warnings nodejs/postject#54

Merged

ovflowd reviewed Oct 19, 2022

View changes

RaisinTen mentioned this pull request Oct 19, 2022

deps: add postject #45066

Closed

RaisinTen mentioned this pull request Oct 20, 2022

Replace commander with parseArgs nodejs/postject#56

Open

RaisinTen force-pushed the add-initial-support-for-single-executable-applications branch from 82806ed to 00de9ff Compare Oct 21, 2022

RaisinTen force-pushed the add-initial-support-for-single-executable-applications branch from 00de9ff to 520c5ab Compare Oct 21, 2022

mhdawson reviewed Oct 21, 2022

View changes

RaisinTen mentioned this pull request Oct 25, 2022

Add a fuse to speed up resource access nodejs/postject#58

Open

mcollina reviewed Oct 26, 2022

View changes

deps/postject/src/dist/postject-api.h Outdated Show resolved Hide resolved

RaisinTen added a commit to nodejs/postject that referenced this pull request Oct 28, 2022

chore: remove Postman copyright from postject-api.h

e3316d9

No need for it. Refs: nodejs/node#45038 (comment) Signed-off-by: Darshan Sen <raisinten@gmail.com>

RaisinTen mentioned this pull request Oct 28, 2022

chore: remove Postman copyright from postject-api.h nodejs/postject#60

Merged

RaisinTen mentioned this pull request Oct 31, 2022

Internally use objcopy --add-section on macOS and Linux nodejs/postject#63

Closed

RaisinTen force-pushed the add-initial-support-for-single-executable-applications branch from 520c5ab to 918a67b Compare Oct 31, 2022

RaisinTen commented Oct 31, 2022

View changes

doc: move stability line after source_link comment

d25924c

Refs: nodejs#45038 (comment) Signed-off-by: Darshan Sen <raisinten@gmail.com>

RaisinTen mentioned this pull request Nov 3, 2022

test: add postject to fixtures #45298

Merged

Trott force-pushed the main branch from 2d76238 to ca3ed36 Compare Nov 12, 2022

Nov	DEC	Jan
	16
2021	2022	2023

src: add initial support for single executable applications #45038

src: add initial support for single executable applications #45038

RaisinTen commented Oct 17, 2022 •

edited

nodejs-github-bot commented Oct 17, 2022

jviotti commented Oct 17, 2022

kapv89 commented Oct 17, 2022 •

edited

jasnell commented Oct 17, 2022

GeoffreyBooth commented Oct 17, 2022

ovflowd commented Oct 19, 2022 •

edited

ovflowd commented Oct 19, 2022

ovflowd left a comment

RaisinTen commented Oct 19, 2022

RaisinTen commented Oct 19, 2022

RaisinTen commented Oct 19, 2022

ovflowd commented Oct 19, 2022 •

edited

RaisinTen commented Oct 19, 2022

mhdawson commented Oct 20, 2022

arcanis commented Oct 21, 2022

RaisinTen commented Oct 21, 2022

mhdawson Oct 21, 2022

mcollina Oct 26, 2022

RaisinTen Oct 31, 2022

mhdawson Oct 21, 2022 •

edited

RaisinTen Oct 31, 2022

mhdawson Oct 21, 2022 •

edited

RaisinTen Oct 31, 2022

pecampelo Nov 1, 2022

ovflowd Nov 1, 2022

RaisinTen Nov 2, 2022 •

edited

src: add initial support for single executable applications #45038

Are you sure you want to change the base?

src: add initial support for single executable applications #45038

Conversation

RaisinTen commented Oct 17, 2022 • edited

nodejs-github-bot commented Oct 17, 2022

jviotti commented Oct 17, 2022

kapv89 commented Oct 17, 2022 • edited

jasnell commented Oct 17, 2022

GeoffreyBooth commented Oct 17, 2022

ovflowd commented Oct 19, 2022 • edited

ovflowd commented Oct 19, 2022

ovflowd left a comment

RaisinTen commented Oct 19, 2022

RaisinTen commented Oct 19, 2022

RaisinTen commented Oct 19, 2022

ovflowd commented Oct 19, 2022 • edited

RaisinTen commented Oct 19, 2022

mhdawson commented Oct 20, 2022

arcanis commented Oct 21, 2022

RaisinTen commented Oct 21, 2022

mhdawson Oct 21, 2022

Choose a reason for hiding this comment

mcollina Oct 26, 2022

Choose a reason for hiding this comment

RaisinTen Oct 31, 2022

Choose a reason for hiding this comment

mhdawson Oct 21, 2022 • edited

Choose a reason for hiding this comment

RaisinTen Oct 31, 2022

Choose a reason for hiding this comment

mhdawson Oct 21, 2022 • edited

Choose a reason for hiding this comment

RaisinTen Oct 31, 2022

Choose a reason for hiding this comment

pecampelo Nov 1, 2022

Choose a reason for hiding this comment

ovflowd Nov 1, 2022

Choose a reason for hiding this comment

RaisinTen Nov 2, 2022 • edited

Choose a reason for hiding this comment

RaisinTen commented Oct 17, 2022 •

edited

kapv89 commented Oct 17, 2022 •

edited

ovflowd commented Oct 19, 2022 •

edited

ovflowd commented Oct 19, 2022 •

edited

mhdawson Oct 21, 2022 •

edited

mhdawson Oct 21, 2022 •

edited

RaisinTen Nov 2, 2022 •

edited