Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Separate morpheus data dir #120

Closed
dagardner-nv opened this issue May 23, 2022 · 0 comments · Fixed by #130
Closed

[FEA] Separate morpheus data dir #120

dagardner-nv opened this issue May 23, 2022 · 0 comments · Fixed by #130
Assignees
@dagardner-nv
Labels
feature request New feature or request

Comments

@dagardner-nv
Copy link
Contributor

dagardner-nv commented May 23, 2022 •
edited
Loading

Is your feature request related to a problem? Please describe.
Currently the Morpheus data dir is 270MB in size.
Some of the files in the directory (labels_nlp.txt, columns_fil.txt, columns_ae.txt & bert-base-cased-hash.txt) are quite small and are used as default values for command line flags.

The files that are used as default values in the morpheus cli need to be copied when we perform a morpheus install (see #60).

Describe the solution you'd like
The smaller txt files should remain in the data dir. The larger jsonlines and csv files should be moved to somewhere under the models dir possibly models/datasets.
@dagardner-nv dagardner-nv added feature request New feature or request Needs Triage Need team to review and classify labels May 23, 2022
@mdemoret-nv mdemoret-nv removed the Needs Triage Need team to review and classify label May 23, 2022
@dagardner-nv dagardner-nv mentioned this issue May 27, 2022
Merged
@dagardner-nv dagardner-nv mentioned this issue Jun 2, 2022
Merged
@ghost ghost closed this as completed in #130 Jun 7, 2022
ghost pushed a commit that referenced this issue Jun 7, 2022
@dagardner-nv
Split data dir, moving large files into examples/data (#130)
b94bd9b 
Brings directory size down to 1.6MB down from 270MB

+ Move large files to `exaples/data` 
+ Migrate existing files in `exaples/data` to git-lfs
+ Add missing `email_with_addresses.jsonlines` needed for phishing detection developer guide
+ Update file paths in docs for data files
+ Update import paths in developer guide

Depends on changes in ~~#62~~
Fixes #120

Authors:
  - David Gardner (https://github.com/dagardner-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - https://github.com/raykallen
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #130
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dagardner-nv dagardner-nv

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Split data dir, moving large files into examples/data dagardner-nv/Morpheus
2 participants
@mdemoret-nv @dagardner-nv