Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evaluate other storage formats for dependency graph. #41

Open
jakemac53 opened this issue Feb 9, 2016 · 3 comments
Open

Evaluate other storage formats for dependency graph. #41

jakemac53 opened this issue Feb 9, 2016 · 3 comments
Labels
P3 A lower priority bug or feature request package:build_runner type-enhancement A request for a change that isn't a bug type-performance
Milestone
M?: Future priority

Comments

@jakemac53
Copy link
Contributor

Will start with JSON just to get things up and running, but proto etc should be evaluated.
@jakemac53 jakemac53 added this to the v0.1.3 release milestone Feb 9, 2016
@jakemac53 jakemac53 removed this from the v0.1.3 release milestone Feb 16, 2016
@jakemac53 jakemac53 mentioned this issue Feb 16, 2016
Merged
@kevmoo kevmoo added type-enhancement A request for a change that isn't a bug and removed enhancement labels Sep 14, 2016
@jakemac53 jakemac53 added P3 A lower priority bug or feature request package:build_runner S3 low labels Jan 30, 2017
@matanlurey matanlurey mentioned this issue Dec 24, 2017
Open
@matanlurey matanlurey added this to the M1: Future priority milestone Dec 24, 2017
@natebosch
Copy link
Member

natebosch commented Jan 19, 2018 •
edited
Loading

msgpack is super easy to drop in to replace JSON.

On e2e_example I didn't see a performance difference and file size dropped from 1.5M to 1.1M.
On angular_components_example I saw (sample size 2) read time drop from 1.7s to 1.5s, and write time drop from 350-430ms to 250-275ms. File size dropped from 8M to 4.9M

The package looks like it might be unmaintained though.
Most recent publish is almost 2 years old, but there are more recent commits.

natebosch added a commit that referenced this issue Jan 19, 2018
@natebosch
Serialize AssetGraph to/from List<int>
0285b99 
Helpful for #41

Centralizes all references to `JSON` within the serialization so that we
can update a single place to experiment with other serialization
formats.
@natebosch natebosch mentioned this issue Jan 19, 2018
Merged
@natebosch
Copy link
Member

The JSON file gzips very well - <10% of it's original size. There is maybe a slight regression on serialize/deserialize time though.

@jakemac53
Copy link
Contributor Author

msgpack sounds interesting as an easy alternative

natebosch added a commit that referenced this issue Jan 19, 2018
@natebosch
Serialize AssetGraph to/from List<int> (#873)
843231f 
Helpful for #41

Centralizes all references to `JSON` within the serialization so that we
can update a single place to experiment with other serialization
formats.
natebosch added a commit that referenced this issue Jan 19, 2018
@natebosch
Serialization optimizations
03a4523 
Towards #41

- Avoid nested lists and serializing ints representing indexes (that
  match the index in the outer list already).
- Avoid repeeating package name strings by storing them in a separate
  list and storing the index into that list.
- For Maps where we don't care about index order switch to HashMap
- Use `growable: false` where possible
- Avoid `.toList()` where possible

This reduces filesize for `asset_components_example` by ~4% and write
time by ~9% and these optimizations also seem to help if we switch to
msgpack.
@natebosch natebosch mentioned this issue Jan 19, 2018
Merged
natebosch added a commit that referenced this issue Jan 19, 2018
@natebosch
Serialization optimizations (#876)
bc5a539 
Towards #41

- Avoid nested lists and serializing ints representing indexes (that
  match the index in the outer list already).
- Avoid repeeating package name strings by storing them in a separate
  list and storing the index into that list.
- For Maps where we don't care about index order switch to HashMap
- Use `growable: false` where possible
- Avoid `.toList()` where possible

This reduces filesize for `asset_components_example` by ~4% and write
time by ~9% and these optimizations also seem to help if we switch to
msgpack.
@MonsterKris MonsterKris mentioned this issue Jan 25, 2019
Closed
@vsmenon vsmenon mentioned this issue Feb 7, 2019
Closed
@corwinsheahan-wf corwinsheahan-wf mentioned this issue Jun 14, 2019
Closed
@jakemac53 jakemac53 mentioned this issue Jul 7, 2020
Closed
natebosch added a commit that referenced this issue Aug 3, 2020
@natebosch
Try out using msgpack for asset graph
b81f439 
Towards #41

See if this makes a meaningful difference for performance.
@natebosch natebosch mentioned this issue Aug 3, 2020
Closed
@annagrin annagrin mentioned this issue Aug 27, 2021
Closed
@yuchuan08 yuchuan08 mentioned this issue Oct 21, 2021
Closed
@Parsaloi Parsaloi mentioned this issue Nov 8, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P3 A lower priority bug or feature request package:build_runner type-enhancement A request for a change that isn't a bug type-performance
Projects
None yet
Milestone
M?: Future priority
Development

No branches or pull requests
4 participants
@kevmoo @matanlurey @jakemac53 @natebosch