X Tutup
The Wayback Machine - https://web.archive.org/web/20230106131251/https://github.com/nodejs/node/pull/45803
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deps: add simdutf dependency #45803

Closed
wants to merge 3 commits into from
Closed

deps: add simdutf dependency #45803

wants to merge 3 commits into from
+30,677 −9

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Dec 9, 2022 •
edited

simdutf provides a faster way of providing utf8 operations with SIMD instructions. @nodejs/undici team was looking for a way to validate utf8 input, and this dependency can make it happen.

Edit: I'm proposing either exposing the following functionality through a new module (like node:encoding) or through util.types or buffer

  • validate_ascii(string)
  • validate_utf8(string)
  • count_utf8(string)

PS: simdutf supports more features, and depending on the need, it makes more sense to expose them through a new module, instead of util.types or buffer.

@nodejs-github-bot
Copy link
Contributor

nodejs-github-bot commented Dec 9, 2022

Review requested:

@nodejs-github-bot nodejs-github-bot added build Issues and PRs related to build files or the CI. dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. tools Issues and PRs related to the tools directory. labels Dec 9, 2022
@anonrig anonrig mentioned this pull request Dec 9, 2022
Merged
12 tasks
@anonrig anonrig force-pushed the deps/simdutf branch 2 times, most recently from 2c20c9a to 6daa546 Compare Dec 9, 2022
@anonrig anonrig changed the title dep: add simdutf dependency deps: add simdutf dependency Dec 9, 2022
@KhafraDev
Copy link
Member

KhafraDev commented Dec 9, 2022

This would help speedup both ws and undici's WebSocket implementation (which is still WIP). When we receive a text frame or receive a close frame with a reason, we need to validate that the buffer contains valid utf-8.

There are a few ways of doing so currently: a js implementation by default in both undici and ws, and optionally a package such as utf-8-validate. Note that simdutf is many times faster than the c++ version of utf-8-validate in the benchmark above, and the js fallback version is the slowest.

Here is a PR from @lpinca that shows massive speedups when using simdutf: websockets/utf-8-validate#101. Considering how widespread usage of ws is, exposing a very fast ability to validate utf-8 would improve a ton of the ecosystem.

@anonrig anonrig force-pushed the deps/simdutf branch 2 times, most recently from 5027cae to e94ba5f Compare Dec 9, 2022
@richardlau richardlau added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 9, 2022
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 9, 2022
@nodejs-github-bot
Copy link
Contributor

nodejs-github-bot commented Dec 9, 2022

CI: https://ci.nodejs.org/job/node-test-pull-request/48372/

@anonrig anonrig force-pushed the deps/simdutf branch 2 times, most recently from bed88cc to 4269faf Compare Dec 9, 2022
mscdex
mscdex reviewed Dec 9, 2022
View changes
deps/simdutf/simdutf.gyp Outdated Show resolved Hide resolved
@github-actions github-actions bot mentioned this pull request Dec 10, 2022
Open
9 tasks
@anonrig anonrig force-pushed the deps/simdutf branch 3 times, most recently from ced7ef2 to 5566c99 Compare Dec 10, 2022
@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@nodejs-github-bot
Copy link
Contributor

nodejs-github-bot commented Dec 10, 2022

CI: https://ci.nodejs.org/job/node-test-pull-request/48377/

@anonrig anonrig added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 10, 2022
@lemire
Copy link
Contributor

lemire commented Dec 22, 2022

@lpinca Please open an issue upstream (simdutf) if you'd like. You have 1.3 million downloads a week, so maybe we can do something specific.

@lpinca
Copy link
Member

lpinca commented Dec 22, 2022 •
edited

I will do it tomorrow. Anway as written in #45803 (comment) those 1.3 million downloads a week are basically all ws downloads. There is little to no usage outside of ws.

@github-actions github-actions bot mentioned this pull request Dec 23, 2022
Open
13 tasks
@lpinca lpinca mentioned this pull request Dec 23, 2022
Closed
This was referenced Dec 24, 2022
Open
Open
Open
targos pushed a commit that referenced this pull request Jan 1, 2023
@anonrig @targos
deps: add simdutf dependency
2277139 
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
targos pushed a commit that referenced this pull request Jan 1, 2023
@anonrig @targos
util: add fast path for text-decoder fatal flag
8ee1d5d 
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
targos pushed a commit that referenced this pull request Jan 1, 2023
@anonrig @lemire @targos
deps: disable avx512 for simutf on benchmark ci
8db4730 
Co-authored-by: Daniel Lemire <daniel@lemire.me>
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
01e1375 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: TBD
@RafaelGSS RafaelGSS mentioned this pull request Jan 2, 2023
Merged
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
fd77454 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
79cf6bc 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
85a26c6 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 2, 2023
@RafaelGSS
2023-01-03, Version 19.4.0 (Current)
3d55e4b 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 3, 2023
@RafaelGSS
2023-01-04, Version 19.4.0 (Current)
ef55c9c 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @RafaelGSS
deps: add simdutf dependency
f537eac 
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @RafaelGSS
util: add fast path for text-decoder fatal flag
73209f0 
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS pushed a commit that referenced this pull request Jan 4, 2023
@anonrig @lemire @RafaelGSS
deps: disable avx512 for simutf on benchmark ci
1f889ae 
Co-authored-by: Daniel Lemire <daniel@lemire.me>
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
980a9ca 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
6e0a7be 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
deps:
  * disable avx512 for simutf on benchmark ci (Yagiz Nizipli) #45803
  * add simdutf dependency (Yagiz Nizipli) #45803
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
1ed0ef2 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 4, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
790cc5b 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @RafaelGSS
deps: add simdutf dependency
344c5ec 
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @RafaelGSS
util: add fast path for text-decoder fatal flag
ae842a4 
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS pushed a commit that referenced this pull request Jan 5, 2023
@anonrig @lemire @RafaelGSS
deps: disable avx512 for simutf on benchmark ci
98fc94a 
Co-authored-by: Daniel Lemire <daniel@lemire.me>
PR-URL: #45803
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-05, Version 19.4.0 (Current)
b181545 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 5, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
e58ed6d 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
RafaelGSS added a commit that referenced this pull request Jan 6, 2023
@RafaelGSS
2023-01-06, Version 19.4.0 (Current)
22a2ec6 
Notable changes:

buffer:
  * (SEMVER-MINOR) add buffer.isUtf8 for utf8 validation (Yagiz Nizipli) #45947
http:
  * (SEMVER-MINOR) improved timeout defaults handling (Paolo Insogna) #45778
net
  * add autoSelectFamily global getter and setter (Paolo Insogna) #45777
os:
  * (SEMVER-MINOR) add availableParallelism() (Colin Ihrig) #45895
util:
  * add fast path for text-decoder fatal flag (Yagiz Nizipli) #45803

PR-URL: #46061
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lemire lemire

@lpinca lpinca

@targos targos

@Uzlopak Uzlopak

@mcollina mcollina

@addaleax addaleax

@mhdawson mhdawson

@mscdex mscdex

@ronag ronag

@daeyeon daeyeon

@bnoordhuis bnoordhuis

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. build Issues and PRs related to build files or the CI. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. dependencies Pull requests that update a dependency file. needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs. performance Issues and PRs related to the performance of Node.js. review wanted PRs that need reviews. tools Issues and PRs related to the tools directory.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
X Tutup