Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA]: Log parsing training for variable token lengths #554

Closed
2 tasks done
raykallen opened this issue Dec 15, 2022 · 0 comments · Fixed by #597
Closed
2 tasks done

[FEA]: Log parsing training for variable token lengths #554

raykallen opened this issue Dec 15, 2022 · 0 comments · Fixed by #597
Assignees
@raykallen
Labels
feature request New feature or request

Comments

@raykallen
Copy link
Contributor

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Critical (currently preventing usage)

Please provide a clear description of problem this feature solves

I want Morpheus log parsing training to have desired token length for the trained model as a variable.

Describe your ideal solution

The main functionality missing is that the token labels created from parsed logs in the training dataset need to be apply to the log pieces that are created when the length of the model is shorter than the length of the logs.

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
  • I have searched the open feature requests and have found no duplicates for this feature request
@raykallen raykallen added the feature request New feature or request label Dec 15, 2022
@raykallen raykallen self-assigned this Dec 15, 2022
@raykallen raykallen mentioned this issue Jan 5, 2023
Merged
@ghost ghost closed this as completed in #597 Jan 23, 2023
ghost pushed a commit that referenced this issue Jan 23, 2023
@raykallen
Log parsing model training for split logs and smaller models (#597)
a220bc0 
added functionality for splitting logs during training to accommodate models with shorter max sequence lengths

closes #554

Authors:
  - https://github.com/raykallen

Approvers:
  - Tad ZeMicheal (https://github.com/tzemicheal)

URL: #597
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raykallen raykallen

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Log parsing model training for split logs and smaller models raykallen/Morpheus
1 participant
@raykallen