Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite zip extraction code to use Buffer in upload.socketRoutes.ts #691

Closed
greenscientist opened this issue Jul 25, 2023 · 4 comments
Closed

Comments

@greenscientist
Copy link
Collaborator

The code in packages/transition-backend/src/api/upload.socketRoutes.ts extract the zip file by read all the content into a variable in memory and then writing the content in a file.
We should be using buffers in this case, to not reach the variable size limit and save some memory.
We could also switch to a package which does it completely transparently.

@greenscientist
Copy link
Collaborator Author

@davidmurray
Copy link
Collaborator

@greenscientist What if we instead just used the standard zip and unzip unix commands? All we are really doing in the code is zipping a directory or unzipping an archive to a directory. Nothing that can't be done with a one line zip or unzip call.
Plus, we would drop the dependency on a library, JSZip.
What do you think? cc @tahini @kaligrafy

@greenscientist
Copy link
Collaborator Author

@davidmurray, we usually don't know what is installed on the system. Also, the code can run on any OSes. We sould need to add this as a requirement in the installation.
the zip algoritgm is simple enough that it's easy to use in via a library.

@davidmurray
Copy link
Collaborator

good point
I'll rewrite the code to use https://www.npmjs.com/package/node-stream-zip then

davidmurray added a commit to davidmurray/transition that referenced this issue Nov 9, 2023
This does it using a stream rather than loading all files into RAM at once. Fixes chairemobilite#691
davidmurray added a commit to davidmurray/transition that referenced this issue Nov 9, 2023
This does it using a stream rather than loading all files into RAM at once. Fixes chairemobilite#691
tahini pushed a commit to tahini/transition that referenced this issue Dec 4, 2023
This does it using a stream rather than loading all files into RAM at once. Fixes chairemobilite#691
davidmurray added a commit to davidmurray/transition that referenced this issue Dec 4, 2023
This does it using a stream rather than loading all files into RAM at once. Fixes chairemobilite#691
@tahini tahini closed this as completed in 80b6eb5 Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants