X Tutup
The Wayback Machine - https://web.archive.org/web/20220321210522/https://github.com/nodejs/node/pull/41681
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream: add asIndexedPairs #41681

Closed

Conversation

benjamingr
Copy link
Member

@benjamingr benjamingr commented Jan 24, 2022

Adds the next (and second to last since we already have .from!) iterator helper from iterator helpers (ref: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs )

I am not basing this on reduce and I'll just deal with merge conflicts. I took Antoine's advice and wrote the tests in .mjs and it is indeed a lot neater although admittedly this is one of the simpler ones.

After this one (and .find which I want to think a bit more about since we may want to support concurrency I want to focus on gathering feedback and iterating on the docs and on the tests (we still need a lot more coverage).

cc @aduh95 @ronag @mcollina @Mesteery @vweevers @VoltrexMaster @nodejs/streams

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 24, 2022

Review requested:

ronag
ronag approved these changes Jan 24, 2022
Copy link
Member

@mcollina mcollina left a comment

lgtm

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 25, 2022

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 25, 2022

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

@nodejs/build this (and other PRs) are failing for a while for an unrelated test on windows.

@Trott
Copy link
Member

@Trott Trott commented Jan 26, 2022

@nodejs/build this (and other PRs) are failing for a while for an unrelated test on windows.

This seems to be affecting only streams PRs, no? How confident are you that the failure isn't related?

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

@Trott it's a test on windows that doesn't use streams or promises. Maybe @targos who wrote the test knows?

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@Trott
Copy link
Member

@Trott Trott commented Jan 26, 2022

Surprisingly (and IMO problematically), it appears that Jenkins does not rebase the pull request against the branch to which it is landing. You might try rebasing, force-pushing, and running CI again, in case the test was fixed by a commit that's landed in the last 48 hours or something like that.

@Trott
Copy link
Member

@Trott Trott commented Jan 26, 2022

Surprisingly (and IMO problematically), it appears that Jenkins does not rebase the pull request against the branch to which it is landing. You might try rebasing, force-pushing, and running CI again, in case the test was fixed by a commit that's landed in the last 48 hours or something like that.

Another possibility might be putting origin/master in the rebase-onto field in Jenkins and not relying on the default.

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

Surprisingly (and IMO problematically), it appears that Jenkins does not rebase the pull request against the branch to which it is landing. You might try rebasing, force-pushing, and running CI again, in case the test was fixed by a commit that's landed in the last 48 hours or something like that.

:O that's... very surprising thanks.

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

So the option in Jenkins is ignored?

image

@richardlau
Copy link
Member

@richardlau richardlau commented Jan 26, 2022

Surprisingly (and IMO problematically), it appears that Jenkins does not rebase the pull request against the branch to which it is landing. You might try rebasing, force-pushing, and running CI again, in case the test was fixed by a commit that's landed in the last 48 hours or something like that.

Eh what? There's definitely code in the CI jobs to do the rebasing. It won't re-rebase if you resume a job though.

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

CI: https://ci.nodejs.org/job/node-test-pull-request/42169/

Edit: please do not abort me! I wanna see if that's the issue

This one is without a rebase just another request-ci

@aduh95 aduh95 removed the request-ci label Jan 26, 2022
@benjamingr benjamingr force-pushed the add-stream-asIndexedPairs branch from 6891680 to f95aa85 Compare Jan 26, 2022
@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

Now another test is failing:

AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Command failed: "C:\workspace\node-test-binary-windows-js-suites\node\Release\node.exe" "C:\workspace\node-test-binary-windows-js-suites\node\test\fixtures\print-chars-from-buffer.js" 1048576 > "C:\workspace\node-test-binary-windows-js-suites\node\test.tmp.571\stdout.txt"
Access is denied.

https://ci.nodejs.org/job/node-test-binary-windows-js-suites/13335/RUN_SUBSET=2,nodes=win10-COMPILED_BY-vs2019/testReport/junit/(root)/test/parallel_test_stdout_to_file/

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

Output of stdout from failure:

<--- Last few GCs --->

[8412:0000015ECFDE8D30]       20 ms: Mark-sweep 2.6 (4.9) -> 2.5 (4.9) MB, 2.5 / 0.0 ms  (average mu = 0.652, current mu = 0.652) allocation failure GC in old space requested
[8412:0000015ECFDE8D30]       26 ms: Mark-sweep 2.8 (5.1) -> 2.8 (5.4) MB, 3.7 / 0.0 ms  (average mu = 0.525, current mu = 0.374) allocation failure GC in old space requested


<--- JS stacktrace --->


#
# Fatal javascript OOM in Reached heap limit
#


node:assert:123
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Wrong exit code of 2147483651! Expected 134 for abort
    at C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-windows-failed-heap-allocation.js:29:10
    at C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:401:15
    at ChildProcess.exithandler (node:child_process:395:5)
    at ChildProcess.emit (node:events:522:28)
    at maybeClose (node:internal/child_process:1090:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: 2147483651,
  expected: 134,
  operator: 'strictEqual'
}

Node.js v18.0.0-pre

It looks like v8 does return this exit code on OOM and Node doesn't get to intercept it - investigating.

Like NearHeapLimitCallback isn't called and V8 exits immediately because the memory grows too quickly but only on windows.

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 26, 2022

This is pretty easy to reproduce by increasing the memory pressure, if I increase the heap size I get 134 consistently but without it I get SIGTRAP pretty consistently.

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 26, 2022

@Trott
Copy link
Member

@Trott Trott commented Jan 26, 2022

Eh what? There's definitely code in the CI jobs to do the rebasing. It won't re-rebase if you resume a job though.

Oh, maybe that was the source of my mistake.

@Trott
Copy link
Member

@Trott Trott commented Jan 26, 2022

This is pretty easy to reproduce by increasing the memory pressure, if I increase the heap size I get 134 consistently but without it I get SIGTRAP pretty consistently.

Does it make sense to move the test from parallel to sequential in that case to make sure it's not competing with other processes for resources?

@Trott
Copy link
Member

@Trott Trott commented Jan 26, 2022

This is pretty easy to reproduce by increasing the memory pressure, if I increase the heap size I get 134 consistently but without it I get SIGTRAP pretty consistently.

Does it make sense to move the test from parallel to sequential in that case to make sure it's not competing with other processes for resources?

Although I guess it raises the question: SHOULD it be sensitive to memory pressure?

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 27, 2022

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 27, 2022

Does it make sense to move the test from parallel to sequential in that case to make sure it's not competing with other processes for resources?

I think just not synchronously causing an OOM in V8 so that it calls the NearHeapLimit thing should be enough?

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 27, 2022

@nodejs-github-bot
Copy link
Contributor

@nodejs-github-bot nodejs-github-bot commented Jan 27, 2022

@benjamingr
Copy link
Member Author

@benjamingr benjamingr commented Jan 27, 2022

Landed in c15639d...7752eed 🎉

@benjamingr benjamingr closed this Jan 27, 2022
benjamingr added a commit that referenced this issue Jan 27, 2022
Add the asIndexedPairs method for readable streams.

PR-URL: #41681
Refs: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
benjamingr added a commit that referenced this issue Jan 27, 2022
The OOM test uses a value that caused an OOM crash from V8 on certain
machines when V8 did not notify the host of OOM soon enough.

PR-URL: #41681
Refs: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Linkgoron added a commit to Linkgoron/node that referenced this issue Jan 31, 2022
Add the asIndexedPairs method for readable streams.

PR-URL: nodejs#41681
Refs: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Linkgoron added a commit to Linkgoron/node that referenced this issue Jan 31, 2022
The OOM test uses a value that caused an OOM crash from V8 on certain
machines when V8 did not notify the host of OOM soon enough.

PR-URL: nodejs#41681
Refs: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
ruyadorno added a commit that referenced this issue Feb 8, 2022
Add the asIndexedPairs method for readable streams.

PR-URL: #41681
Refs: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
ruyadorno added a commit that referenced this issue Feb 8, 2022
The OOM test uses a value that caused an OOM crash from V8 on certain
machines when V8 did not notify the host of OOM soon enough.

PR-URL: #41681
Refs: https://github.com/tc39/proposal-iterator-helpers#asindexedpairs
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
X Tutup