Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buffer: add Buffer.copyFrom(...) #46500

Closed
wants to merge 1 commit into from

Conversation

jasnell
Copy link
Member

@jasnell jasnell commented Feb 4, 2023

This is a variation on Buffer.from(arrayBufferView) that copies the bytes and does not truncate when using a multibyte TypedArray. For instance,

Buffer.from(new Uint16Array([0xffff]))
<Buffer ff>

Buffer.copyBytesFrom(new Uint16Array([0xffff]))
<Buffer ff ff> 

Open to suggestions on a better name.

Fixes: #43862

@jasnell jasnell added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 4, 2023
@nodejs-github-bot nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. needs-ci PRs that need a full CI run. labels Feb 4, 2023
lib/buffer.js Outdated Show resolved Hide resolved
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 4, 2023
@nodejs-github-bot

This comment was marked as outdated.

lib/buffer.js Outdated Show resolved Hide resolved
lib/buffer.js Outdated Show resolved Hide resolved
lib/buffer.js Outdated Show resolved Hide resolved
lib/buffer.js Outdated Show resolved Hide resolved
doc/api/buffer.md Outdated Show resolved Hide resolved
doc/api/buffer.md Outdated Show resolved Hide resolved
doc/api/buffer.md Outdated Show resolved Hide resolved
lib/buffer.js Outdated Show resolved Hide resolved
doc/api/buffer.md Outdated Show resolved Hide resolved
@addaleax
Copy link
Member

addaleax commented Feb 5, 2023

Counter-suggestion: Mark Buffer as a legacy API? There's no good reason for not using standardized equivalents in 2023.

@jasnell
Copy link
Member Author

jasnell commented Feb 5, 2023

Mark Buffer as a legacy API? There's no good reason for not using standardized equivalents in 2023.

As much as I do prefer standardized APIs, Buffer clearly is never ever going to decline in use and is being actively implemented and used in at least three other runtimes right now. It's not going anywhere and there's really no benefit I think to changing it's status

@LiviaMedeiros
Copy link
Contributor

Is there a reason to work with TypedArray and not ArrayBufferView (which also includes DataView)?

TypedArray.from is basically a variant of Array.from and therefore requires iterable value as first parameter, but it should be irrelevant for this method.

@jasnell jasnell force-pushed the buffer-fromtypedarray branch 2 times, most recently from 9b2f1e7 to ce45b9f Compare February 5, 2023 15:55
@jasnell
Copy link
Member Author

jasnell commented Feb 5, 2023

@LiviaMedeiros ... the implementation does use ArrayBufferView. I've updated the doc change to reflect that.

@jasnell jasnell changed the title buffer: add Buffer.fromTypedArray(...) buffer: add Buffer.copyFrom(...) Feb 5, 2023
@jasnell jasnell added semver-minor PRs that contain new features and should be released in the next minor version. request-ci Add this label to start a Jenkins CI on a PR. labels Feb 5, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 5, 2023
@nodejs-github-bot

This comment was marked as outdated.

@aduh95
Copy link
Contributor

aduh95 commented Feb 5, 2023

@LiviaMedeiros ... the implementation does use ArrayBufferView. I've updated the doc change to reflect that.

How do we interpret length for DataViews? As the number of 8-bit chunks? Or 32-bit chunks? Same questions for offset. Whatever we chose, it would feel quite arbitrary to me, I’d rather have us only support TypedArray, and maybe DataView only if length and offset are not supplied.

@jasnell
Copy link
Member Author

jasnell commented Feb 5, 2023

Yeah, I realized that the DataView wouldn't work with the current implementation anyway. Let's limit it to TypedArray. I don't really like the idea of supporting DataView only part of the time, and would rather just make it consistent no matter what other arguments are passed in.

@jasnell jasnell added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 5, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 5, 2023
@nodejs-github-bot
Copy link
Collaborator

@LiviaMedeiros
Copy link
Contributor

How do we interpret length for DataViews? As the number of 8-bit chunks? Or 32-bit chunks? Same questions for offset.

It might be reasonable to interpret both as number of bytes, and length can become byteLength.
This would also remove limitation of having to copy full chunks.

I'm not quite sure if there are usecases of copy where counting elements is preferred over counting bytes and at the same time it wouldn't be better to process elements on copy (i.e. by using TypedArray.from(arrayLike, mapFn) which also can be ported to Buffer).

lib/buffer.js Outdated Show resolved Hide resolved
lib/buffer.js Show resolved Hide resolved
doc/api/buffer.md Outdated Show resolved Hide resolved
Copy link
Contributor

@aduh95 aduh95 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit message needs to be amended to use the new name.

lib/buffer.js Outdated Show resolved Hide resolved
test/parallel/test-buffer-from.js Outdated Show resolved Hide resolved
test/parallel/test-buffer-from.js Outdated Show resolved Hide resolved
@jasnell jasnell added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Feb 25, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2023
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot
Copy link
Collaborator

nodejs-github-bot commented Feb 26, 2023

jasnell added a commit that referenced this pull request Feb 26, 2023
Fixes: #43862
PR-URL: #46500
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
@jasnell
Copy link
Member Author

jasnell commented Feb 26, 2023

Landed in 311fb04

@jasnell jasnell closed this Feb 26, 2023
targos pushed a commit that referenced this pull request Mar 13, 2023
Fixes: #43862
PR-URL: #46500
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
targos pushed a commit that referenced this pull request Mar 14, 2023
Fixes: #43862
PR-URL: #46500
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
targos added a commit that referenced this pull request Mar 14, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
src:
  * (SEMVER-MINOR) add `fs.openAsBlob` to support File-backed Blobs (James M Snell) #45258
tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
wasi:
  * (SEMVER-MINOR) add support for version when creating WASI (Michael Dawson) #46469
worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47086
targos added a commit that referenced this pull request Mar 14, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
src:
  * (SEMVER-MINOR) add `fs.openAsBlob` to support File-backed Blobs (James M Snell) #45258
tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
wasi:
  * (SEMVER-MINOR) add support for version when creating WASI (Michael Dawson) #46469
worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47087
targos added a commit that referenced this pull request Mar 14, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
src:
  * (SEMVER-MINOR) add `fs.openAsBlob` to support File-backed Blobs (James M Snell) #45258
tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
wasi:
  * (SEMVER-MINOR) add support for version when creating WASI (Michael Dawson) #46469
worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47087
targos added a commit that referenced this pull request Mar 14, 2023
Notable changes:

buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
src:
  * (SEMVER-MINOR) add `fs.openAsBlob` to support File-backed Blobs (James M Snell) #45258
tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
wasi:
  * (SEMVER-MINOR) add support for version when creating WASI (Michael Dawson) #46469
worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47087

```js
const u16 = new Uint16Array([0, 0xffff]);
const buf = Buffer.copyBytesFrom(u16, 0, 1);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jasnell Was the offset in this example supposed to be 1?

danielleadams pushed a commit that referenced this pull request Apr 11, 2023
Fixes: #43862
PR-URL: #46500
Reviewed-By: Yagiz Nizipli <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
danielleadams added a commit that referenced this pull request Apr 11, 2023
Notable changes:

* buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
* doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
  * add debadree25 to collaborators (Debadree Chatterjee) #46716
  * add deokjinkim to collaborators (Deokjin Kim) #46444
* events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
* lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
  * (SEMVER-MINOR) add aborted() utility function (Debadree Chatterjee) #46494
* src:
  * (SEMVER-MINOR) allow optional Isolate termination in node::Stop() (Shelley Vohr) #46583
  * (SEMVER-MINOR) allow embedder control of code generation policy (Shelley Vohr) #46368
* stream:
  * (SEMVER-MINOR) add abort signal for ReadableStream and WritableStream (Debadree Chatterjee) #46273
* tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
* url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
* worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: TBD
danielleadams added a commit that referenced this pull request Apr 11, 2023
Notable changes:

* buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
* doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
  * add debadree25 to collaborators (Debadree Chatterjee) #46716
  * add deokjinkim to collaborators (Deokjin Kim) #46444
* events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
* lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
  * (SEMVER-MINOR) add aborted() utility function (Debadree Chatterjee) #46494
* src:
  * (SEMVER-MINOR) allow optional Isolate termination in node::Stop() (Shelley Vohr) #46583
  * (SEMVER-MINOR) allow embedder control of code generation policy (Shelley Vohr) #46368
* stream:
  * (SEMVER-MINOR) add abort signal for ReadableStream and WritableStream (Debadree Chatterjee) #46273
* tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
* url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
* worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47502
danielleadams added a commit that referenced this pull request Apr 11, 2023
Notable changes:

* buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
* doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
  * add debadree25 to collaborators (Debadree Chatterjee) #46716
  * add deokjinkim to collaborators (Deokjin Kim) #46444
* events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
* lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
  * (SEMVER-MINOR) add aborted() utility function (Debadree Chatterjee) #46494
* src:
  * (SEMVER-MINOR) allow optional Isolate termination in node::Stop() (Shelley Vohr) #46583
  * (SEMVER-MINOR) allow embedder control of code generation policy (Shelley Vohr) #46368
  * (SEMVER-MINOR) add initial support for single executable applications (Darshan Sen) #45038
* stream:
  * (SEMVER-MINOR) add abort signal for ReadableStream and WritableStream (Debadree Chatterjee) #46273
* tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
* url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
* worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47502
danielleadams added a commit that referenced this pull request Apr 12, 2023
Notable changes:

Add initial support for single executable applications

Compile a JavaScript file into a single executable application:

```console
$ echo 'console.log(`Hello, ${process.argv[2]}!`);' > hello.js

$ cp $(command -v node) hello

$ npx postject hello NODE_JS_CODE hello.js \
    --sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2

$ npx postject hello NODE_JS_CODE hello.js \
    --sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2 \
    --macho-segment-name NODE_JS

$ ./hello world
Hello, world!
```

Contributed by Darshan Sen in #45038

Replace url parser with Ada

Node.js gets a new URL parser called Ada that is compliant with the WHATWG
URL Specification and provides more than 100% performance improvement to
the existing implementation.

Contributed by Yagiz Nizipli in #46410

Other notable changes:

* buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
* doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
  * add debadree25 to collaborators (Debadree Chatterjee) #46716
  * add deokjinkim to collaborators (Deokjin Kim) #46444
* events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
* lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
  * (SEMVER-MINOR) add aborted() utility function (Debadree Chatterjee) #46494
* src:
  * (SEMVER-MINOR) allow optional Isolate termination in node::Stop() (Shelley Vohr) #46583
  * (SEMVER-MINOR) allow embedder control of code generation policy (Shelley Vohr) #46368
* stream:
  * (SEMVER-MINOR) add abort signal for ReadableStream and WritableStream (Debadree Chatterjee) #46273
* tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
* url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
* worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47502
danielleadams added a commit that referenced this pull request Apr 13, 2023
Notable changes:

Add initial support for single executable applications

Compile a JavaScript file into a single executable application:

```console
$ echo 'console.log(`Hello, ${process.argv[2]}!`);' > hello.js

$ cp $(command -v node) hello

$ npx postject hello NODE_JS_CODE hello.js \
    --sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2

$ npx postject hello NODE_JS_CODE hello.js \
    --sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2 \
    --macho-segment-name NODE_JS

$ ./hello world
Hello, world!
```

Contributed by Darshan Sen in #45038

Replace url parser with Ada

Node.js gets a new URL parser called Ada that is compliant with the WHATWG
URL Specification and provides more than 100% performance improvement to
the existing implementation.

Contributed by Yagiz Nizipli in #46410

Other notable changes:

* buffer:
  * (SEMVER-MINOR) add Buffer.copyBytesFrom(...) (James M Snell) #46500
* doc:
  * add marco-ippolito to collaborators (Marco Ippolito) #46816
  * add debadree25 to collaborators (Debadree Chatterjee) #46716
  * add deokjinkim to collaborators (Deokjin Kim) #46444
* events:
  * (SEMVER-MINOR) add listener argument to listenerCount (Paolo Insogna) #46523
* lib:
  * (SEMVER-MINOR) add AsyncLocalStorage.bind() and .snapshot() (flakey5) #46387
  * (SEMVER-MINOR) add aborted() utility function (Debadree Chatterjee) #46494
* src:
  * (SEMVER-MINOR) allow optional Isolate termination in node::Stop() (Shelley Vohr) #46583
  * (SEMVER-MINOR) allow embedder control of code generation policy (Shelley Vohr) #46368
* stream:
  * (SEMVER-MINOR) add abort signal for ReadableStream and WritableStream (Debadree Chatterjee) #46273
* tls:
  * (SEMVER-MINOR) support automatic DHE (Tobias Nießen) #46978
* url:
  * (SEMVER-MINOR) implement URLSearchParams size getter (James M Snell) #46308
* worker:
  * (SEMVER-MINOR) add support for worker name in inspector and trace_events (Debadree Chatterjee) #46832

PR-URL: #47502
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. buffer Issues and PRs related to the buffer subsystem. needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Buffer.from(<ArrayBufferView>)
7 participants