X Tutup
The Wayback Machine - https://web.archive.org/web/20201025234145/https://github.com/nodejs/node/pull/35775
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature-request - add [lazy] option `mkdirRecursive` to fs.writeFile[Sync] and fs.appendFile[Sync] #35775

Draft
wants to merge 2 commits into
base: master
from

Conversation

@kaizhu256
Copy link

@kaizhu256 kaizhu256 commented Oct 23, 2020

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

this is proof-of-concept to implement feature-request #33559. if feedback is positive/viable, then i will proceed with adding documentation, tests, and other checklist items.

motivation

this feature is intended to improve ergonomics/simplify-scripting-tasks when:

  • scaffolding new web-projects
  • creating build-artifacts/coverage-files during ci
  • cloning website with web-crawler
  • uploading files to server

in above tasks, user often has no deterministic-knowledge on directory-structure before file-creation.
this feature allows user to lazily create ad-hoc directory-structures as need during file-creation using ergonomic syntax:

fs.writeFileSync(
    "foo/bar/baz/qux.txt",
    "hello world!",
    { mkdirRecursive: true } // will lazily "mkdir -p" foo/bar/baz as needed
);

performance impact and benchmark

the benchmark (in windows 10) comparing this pr-branch against master-branch shows no-performance-impact on fs.writeFile[Sync] or fs.appendFile[Sync] when option { mkdirRecursive: true } is not used.

when option { mkdirRecursive: true } is enabled:

  • fs.writeFileSync and fs.appendFileSync is ~10% slower
    at lazy-adhoc-directory-creation (vs eager-determistic-directory-creation)
  • fs.writeFile and fs.appendFile is ~70% slower
    at lazy-adhoc-directory-creation (vs eager-determistic-directory-creation)

windows benchmark result

the following results should be reproducible by following benchmark instructions at https://github.com/kaizhu256/node/tree/benchmark.fs.writeFile.mkdirRecursive#run-windows-benchmark

async-operation                                         master-branch           pr-branch       performance-impact
30 * 1000 * (fs.writeFile      w/ no-mkdir      )      934 +/-  50 ms      929 +/-  82 ms    no performance-impact
30 * 1000 * (fs.writeFile      w/ fs.mkdir      )      722 +/- 163 ms      722 +/- 159 ms    no performance-impact
30 * 1000 * (fs.writeFile      w/ mkdirRecursive)             ---- ms     1244 +/- 135 ms    72% slower (lazy-mkdirRecursive vs eager-fs.mkdir)

async-operation                                         master-branch           pr-branch       performance-impact
30 * 1000 * (fs.appendFile     w/ no-mkdir      )      983 +/-  62 ms      966 +/-  92 ms    no performance-impact
30 * 1000 * (fs.appendFile     w/ fs.mkdir      )      739 +/- 175 ms      730 +/- 184 ms    no performance-impact
30 * 1000 * (fs.appendFile     w/ mkdirRecursive)             ---- ms     1252 +/- 126 ms    69% slower (lazy-mkdirRecursive vs eager-fs.mkdir)


 sync-operation                                         master-branch           pr-branch       performance-impact
30 * 1000 * (fs.writeFileSync  w/ no-mkdir      )     1008 +/-  42 ms     1007 +/-  45 ms    no performance-impact
30 * 1000 * (fs.writeFileSync  w/ fs.mkdirSync  )     1643 +/- 121 ms     1611 +/- 117 ms    no performance-impact
30 * 1000 * (fs.writeFileSync  w/ mkdirRecursive)             ---- ms     1791 +/-  87 ms     9% slower (lazy-mkdirRecursive vs eager-fs.mkdir)

 sync-operation                                         master-branch           pr-branch       performance-impact
30 * 1000 * (fs.appendFileSync w/ no-mkdir      )      999 +/-  47 ms     1019 +/-  53 ms    no performance-impact
30 * 1000 * (fs.appendFileSync w/ fs.mkdirSync  )     1617 +/- 103 ms     1619 +/- 109 ms    no performance-impact
30 * 1000 * (fs.appendFileSync w/ mkdirRecursive)             ---- ms     1789 +/- 104 ms    11% slower (lazy-mkdirRecursive vs eager-fs.mkdir)

bikeshed of name mkdirRecursive

when scripting the benchmark, i realized the name mkdirRecursive is tedious to type. it also doesn't convey the lazy-nature of this operation, or the -p attribute. am open to other naming suggestions (e.g. { mkdirp: true })

…and fs.writeFile[Sync]
@Trott
Copy link
Member

@Trott Trott commented Oct 23, 2020

Welcome @kaizhu256 and thanks for the pull request. I'm going to put this into Draft mode while it's being discussed. It will need tests and documentation before we can land it.

@Trott Trott marked this pull request as draft Oct 23, 2020
@Trott
Copy link
Member

@Trott Trott commented Oct 24, 2020

@Trott
Copy link
Member

@Trott Trott commented Oct 25, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.
X Tutup