Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration of a static filesystem for the node_modules #47998

Merged
merged 151 commits into from
May 5, 2020
Merged

Integration of a static filesystem for the node_modules #47998

merged 151 commits into from
May 5, 2020

Conversation

mistic
Copy link
Member

@mistic mistic commented Oct 11, 2019 •
edited
Loading

Closes #43357
Closes #17176

That is the so awaited very first integration of a static filesystem for the node_modules on Kibana.
I really have the impression that would be huge, thank you very much for the support team!

== Benefits ==

  • decrease number of files we ship within the distributable (preventing us from create a windows service for Kibana)
  • avoid problems like the max path length of a file (Windows)
  • decrease the amount of time needed for decompress the distributable

== Benchmark ==

Collected when using a static-fs with Kibana for the node_modules.
The hardware used was:

Computer: Custom Windows Box
OS: Windows 10 64bits
Processor: 3,7 GHz AMD Ryzen 2700x
Memory: 16 GB 2933 MHz DDR4
Disk: SSD 500GB NVMe

Stats (on distributable) Without Static-Fs With Static-Fs Difference
Unzip (windows default tool) 47m 8m -83%
Number of files 137476 22803 -83%
Used memory 298MB 335MB +12%
Load time (cold start) ~3000ms 3253ms +8%

Along the normal usage of the application no performance issue was found.

mistic added 30 commits August 13, 2018 17:49
@mistic
feat(NA): added new build step to generate a static file system for n…
e0b1cdc 
…ode_modules and monkey patch the bin start files to include the static file system loader on begining.
@mistic
chore(NA): updated lockfile.
41b785d
@mistic
chore(NA): replace import to the correct static-fs files.
8b104e1
@mistic
chore(NA): solve conflicts merging with master.
084fc7a
@mistic
fix(NA): resolve for a correct static fs loader.
34106f7
@mistic
chore(NA): last static-fs working version.
b60c33d
@mistic
feat(NA): changed patched entryPoints.
956e538
@mistic
chore(NA): update to last static fs.
470e125
@mistic
Merge branch 'master' into static-filesystem
c59aaa8
@mistic
Merge branch 'master' into static-filesystem
6fcd7e5
@mistic
fix(NA): fix imports on webpack shims. chore(NA): added new static-fs…
bc6db67 
… version.
@mistic
chore(NA): update to static-fs with stat and statSync.
64f40b5
@mistic
fix(NA): vendor chunk filtering in order to include default node modu…
319011b 
…les and static node modules.
@mistic
refact(NA): major refact on create static modules fs task.
00cb528
@mistic
Merge branch 'master' into static-filesystem
0654b49
@mistic
fix(NA): remove global from clean empty folders task.
635d82f
@mistic
chore(NA): merge and solve with last master. - UNTESTED
f42c59c
@mistic
Merge branch 'master' into static-filesystem
b56f779
@mistic
chore(NA): change args order for deleteAll.
112f3ec
@mistic
Merge branch 'master' into static-filesystem
7819ca3
@mistic
chore(NA): move to the new static fs
ce1ba90
@mistic
Merge branch 'master' into static-filesystem
6a7a5e8
@mistic
feat(NA): migrate to last apis of static fs
c87057d
@mistic
fix(NA): patch correct entry file.
5d769c3
@mistic
Merge branch 'master' into static-filesystem
4179ac7
@mistic
chore(NA): merge and solve conflicts with master
58faf4e
@mistic
chore(NA): last stable integration with static-fs task
b8e5cfd
@mistic
Merge remote-tracking branch 'upstream/master' into static-filesystem
38724c4
@mistic
chore(NA): rollback some unecessary changes
b0bf35e
@mistic
Merge remote-tracking branch 'upstream/master' into static-filesystem
16f0fa1
@mistic
Copy link
Member Author

mistic commented Apr 27, 2020

@elasticmachine merge upstream

@mistic
Copy link
Member Author

mistic commented Apr 29, 2020

@elasticmachine merge upstream

@mistic
Copy link
Member Author

mistic commented May 5, 2020

@elasticmachine merge upstream

@mistic mistic merged commit 33376fc into elastic:master May 5, 2020
@mistic
Copy link
Member Author

mistic commented May 5, 2020 •
edited
Loading

7.x/7.8.0: 6c732a9

@mistic mistic mentioned this pull request May 5, 2020
Merged
mistic added a commit to mistic/kibana that referenced this pull request May 5, 2020
@mistic @elasticmachine
Integration of a static filesystem for the node_modules (elastic#47998)
e9dc265 
* feat(NA): added new build step to generate a static file system for node_modules and monkey patch the bin start files to include the static file system loader on begining.

* chore(NA): updated lockfile.

* chore(NA): replace import to the correct static-fs files.

* fix(NA): resolve for a correct static fs loader.

* chore(NA): last static-fs working version.

* feat(NA): changed patched entryPoints.

* chore(NA): update to last static fs.

* fix(NA): fix imports on webpack shims. chore(NA): added new static-fs version.

* chore(NA): update to static-fs with stat and statSync.

* fix(NA): vendor chunk filtering in order to include default node modules and static node modules.

* refact(NA): major refact on create static modules fs task.

* fix(NA): remove global from clean empty folders task.

* chore(NA): change args order for deleteAll.

* chore(NA): move to the new static fs

* feat(NA): migrate to last apis of static fs

* fix(NA): patch correct entry file.

* chore(NA): last stable integration with static-fs task

* chore(NA): rollback some unecessary changes

* chore(NA): remove changes on sinon webpackshim

* chore(NA): integrate with official static-fs version

* chore(NA): integrate last thread-loader version

* chore(NA): added last static-fs version

* chore(NA): upgrade to last static-fs version

* chore(NA): upgrade to last static-fs version 1.1.1

* chore(NA): rever static-fs version to 1.1.0

* chore(NA): upgrade static-fs to 1.1.1

* chore(NA): upgrade static-fs for last 1.2.0 version

* chore(NA): update package.json

* chore(NA): run apm after patched environment

* chore(NA): specify rule to disable eslint for

* chore(NA): remove changes in the src/apm file

* chore(NA): change apm order in the dev scripts

* chore(na): update static-fs versin

* chore(NA): bump static fs version

* chore(NA): correctly lint task file

* chore(NA): move away from rimraf to del

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump to last static-fs version

* chore(NA): apply changed according PR review feedback

* chore(NA): remove changes to base optimizer

* chore(NA): fix discover pattern

* chore(NA): bump static-fs version

* chore(NA): bump static fs to 1.6.3

* chore(NA): bump static-fs to 1.6.4

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs for 1.8.3

* chore(NA): bump static-fs to 1.9.0

* chore(NA): bump static-fs to 1.9.1

* chore(NA): update to last static-fs version 1.10.0

* chore(NA): moving to @elastic/static-fs package

* fix(NA): change import to the new package

* chore(NA): bump elastic static-fs to 1.0.1

Co-authored-by: spalger <email@spalger.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@mistic mistic removed the review label May 5, 2020
mistic added a commit that referenced this pull request May 5, 2020
@mistic @elasticmachine
Integration of a static filesystem for the node_modules (#47998) (#65346
6c732a9 
)

* feat(NA): added new build step to generate a static file system for node_modules and monkey patch the bin start files to include the static file system loader on begining.

* chore(NA): updated lockfile.

* chore(NA): replace import to the correct static-fs files.

* fix(NA): resolve for a correct static fs loader.

* chore(NA): last static-fs working version.

* feat(NA): changed patched entryPoints.

* chore(NA): update to last static fs.

* fix(NA): fix imports on webpack shims. chore(NA): added new static-fs version.

* chore(NA): update to static-fs with stat and statSync.

* fix(NA): vendor chunk filtering in order to include default node modules and static node modules.

* refact(NA): major refact on create static modules fs task.

* fix(NA): remove global from clean empty folders task.

* chore(NA): change args order for deleteAll.

* chore(NA): move to the new static fs

* feat(NA): migrate to last apis of static fs

* fix(NA): patch correct entry file.

* chore(NA): last stable integration with static-fs task

* chore(NA): rollback some unecessary changes

* chore(NA): remove changes on sinon webpackshim

* chore(NA): integrate with official static-fs version

* chore(NA): integrate last thread-loader version

* chore(NA): added last static-fs version

* chore(NA): upgrade to last static-fs version

* chore(NA): upgrade to last static-fs version 1.1.1

* chore(NA): rever static-fs version to 1.1.0

* chore(NA): upgrade static-fs to 1.1.1

* chore(NA): upgrade static-fs for last 1.2.0 version

* chore(NA): update package.json

* chore(NA): run apm after patched environment

* chore(NA): specify rule to disable eslint for

* chore(NA): remove changes in the src/apm file

* chore(NA): change apm order in the dev scripts

* chore(na): update static-fs versin

* chore(NA): bump static fs version

* chore(NA): correctly lint task file

* chore(NA): move away from rimraf to del

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump to last static-fs version

* chore(NA): apply changed according PR review feedback

* chore(NA): remove changes to base optimizer

* chore(NA): fix discover pattern

* chore(NA): bump static-fs version

* chore(NA): bump static fs to 1.6.3

* chore(NA): bump static-fs to 1.6.4

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs for 1.8.3

* chore(NA): bump static-fs to 1.9.0

* chore(NA): bump static-fs to 1.9.1

* chore(NA): update to last static-fs version 1.10.0

* chore(NA): moving to @elastic/static-fs package

* fix(NA): change import to the new package

* chore(NA): bump elastic static-fs to 1.0.1

Co-authored-by: spalger <email@spalger.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: spalger <email@spalger.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kobelb added a commit to kobelb/kibana that referenced this pull request May 20, 2020
@kobelb
Revert "Integration of a static filesystem for the node_modules (elas…
5b327ad 
…tic#47998)"

This reverts commit 33376fc.
@kobelb kobelb mentioned this pull request May 20, 2020
Merged
kobelb added a commit that referenced this pull request May 20, 2020
@kobelb
Revert "Integration of a static filesystem for the node_modules (#47998…
0e55b24 
…)" (#67124)

This reverts commit 33376fc.
This was referenced May 20, 2020
Merged
Merged
kobelb added a commit to kobelb/kibana that referenced this pull request May 20, 2020
@kobelb
Revert "Integration of a static filesystem for the node_modules (elas…
ca0fb59 
…tic#47998)" (elastic#67124)

This reverts commit 33376fc.
jloleysens added a commit to jloleysens/kibana that referenced this pull request May 20, 2020
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into index-managem…
d6da4bd 
…ent/add-support-in-url-for-hidden-toggle

* 'master' of github.com:elastic/kibana: (49 commits)
  [Uptime] Improve responsiveness details page (elastic#67034)
  skip flaky suite (elastic#66669)
  Revert "Integration of a static filesystem for the node_modules (elastic#47998)" (elastic#67124)
  Support api_integration/kibana/stats against remote hosts (elastic#53000)
  chore(NA): add module name mapper for src plugins on x-pack (elastic#67103)
  Change the error message on TSVB in order to be more user friendly (elastic#67090)
  [kbn/optimizer] poll parent process to avoid zombie processes (elastic#67059)
  [Visualize] Lazy load default editor, fix duplicated styles (elastic#66732)
  Bump styled-component dependencies (elastic#66611)
  Bump react-markdown dependencies (elastic#66615)
  Fix Core docs links (elastic#66977)
  Timelion graph is not refreshing content after searching or filtering (elastic#67023)
  Remove `--xpack.endpoint.enabled=true` from README.md file (elastic#67053)
  Move apm tutorial from apm plugin into apm_oss plugin (elastic#66432)
  [Logs UI] Restore call to `UsageCollector.countLogs` (elastic#67051)
  Remove unused license check result from LP Security plugin (elastic#66966)
  [Saved Objects] adds support for including hidden types in saved objects client (elastic#66879)
  [Discover] Deangularize timechart header (elastic#66532)
  [Discover] Improve and unskip a11y context view test (elastic#66959)
  [SIEM] Refactor Timeline.timelineType draft to Timeline.status draft (elastic#66864)
  ...

# Conflicts:
#	x-pack/plugins/index_management/__jest__/client_integration/helpers/home.helpers.ts
kobelb added a commit that referenced this pull request May 20, 2020
@kobelb
Revert "Integration of a static filesystem for the node_modules (#47998…
2b515b1 
…)" (#67124) (#67134)

This reverts commit 33376fc.
kobelb added a commit that referenced this pull request May 20, 2020
@kobelb
Revert "Integration of a static filesystem for the node_modules (#47998…
931ed76 
…)" (#67124) (#67135)

This reverts commit 33376fc.
gmmorris added a commit to gmmorris/kibana that referenced this pull request May 21, 2020
@gmmorris
Merge branch 'master' into saved-objects/scoped-client-with-extra-types
0660d44 
* master:
  [apm] Annotation API documentation (elastic#65963)
  [Uptime] Improve responsiveness details page (elastic#67034)
  skip flaky suite (elastic#66669)
  Revert "Integration of a static filesystem for the node_modules (elastic#47998)" (elastic#67124)
  Support api_integration/kibana/stats against remote hosts (elastic#53000)
  chore(NA): add module name mapper for src plugins on x-pack (elastic#67103)
  Change the error message on TSVB in order to be more user friendly (elastic#67090)
  [kbn/optimizer] poll parent process to avoid zombie processes (elastic#67059)
  [Visualize] Lazy load default editor, fix duplicated styles (elastic#66732)
  Bump styled-component dependencies (elastic#66611)
  Bump react-markdown dependencies (elastic#66615)
  Fix Core docs links (elastic#66977)
  Timelion graph is not refreshing content after searching or filtering (elastic#67023)
  Remove `--xpack.endpoint.enabled=true` from README.md file (elastic#67053)
  Move apm tutorial from apm plugin into apm_oss plugin (elastic#66432)
  [Logs UI] Restore call to `UsageCollector.countLogs` (elastic#67051)
  Remove unused license check result from LP Security plugin (elastic#66966)
gmmorris added a commit to gmmorris/kibana that referenced this pull request May 21, 2020
@gmmorris
Merge branch 'master' into actions/hide-action-type
c3c639d 
* master: (21 commits)
  [Alerting] Hides the `alert` SavedObjects type (elastic#66719)
  skip flaky suite (elastic#66869)
  fix visual baseline tests
  [kbn/optimizer] require fsevents on macos (elastic#67147)
  [APM] Fix obscured service map connections (elastic#67129)
  [apm] Annotation API documentation (elastic#65963)
  [Uptime] Improve responsiveness details page (elastic#67034)
  skip flaky suite (elastic#66669)
  Revert "Integration of a static filesystem for the node_modules (elastic#47998)" (elastic#67124)
  Support api_integration/kibana/stats against remote hosts (elastic#53000)
  chore(NA): add module name mapper for src plugins on x-pack (elastic#67103)
  Change the error message on TSVB in order to be more user friendly (elastic#67090)
  [kbn/optimizer] poll parent process to avoid zombie processes (elastic#67059)
  [Visualize] Lazy load default editor, fix duplicated styles (elastic#66732)
  Bump styled-component dependencies (elastic#66611)
  Bump react-markdown dependencies (elastic#66615)
  Fix Core docs links (elastic#66977)
  Timelion graph is not refreshing content after searching or filtering (elastic#67023)
  Remove `--xpack.endpoint.enabled=true` from README.md file (elastic#67053)
  Move apm tutorial from apm plugin into apm_oss plugin (elastic#66432)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@spalger spalger spalger left review comments

@jbudz jbudz jbudz approved these changes

@watson watson watson approved these changes

Assignees
No one assigned
Labels
build release_note:enhancement Team:Operations Team label for Operations Team v7.8.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Integrate a static filesystem for node modules with kibana Investigate slow package installs
8 participants
@mistic @elasticmachine @spalger @tylersmalley @watson @legrego @kibanamachine @jbudz