Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TASK: Cleanup the legacy namespace for lineage event #449

Closed
3 tasks done
csun-cpointe opened this issue Oct 31, 2024 · 1 comment
Closed
3 tasks done

TASK: Cleanup the legacy namespace for lineage event #449

csun-cpointe opened this issue Oct 31, 2024 · 1 comment
Assignees
Labels
slacktask Small ticket done in down time task Support ticket - typically not code related
Milestone

Comments

@csun-cpointe
Copy link
Contributor

csun-cpointe commented Oct 31, 2024

Description

In v1.7.0 we have released the OpenLineage Namesapce Conventions to better follow OpenLineage's guidelines. Moving forward, namespaces should be defined in the data-lineage.properties file. We are cleaning up the data.lineage.namespace properties in a project's data-lineage.properties file, which was supported as a fallback but will no longer be supported in release 1.10

DOD

Acceptance criteria required to complete the work

  • Clean up the data.lineage.namespace support functions and tests
  • Clean up the data.lineage.namespace properties in the data-lineage.properties file
  • The pyspark, spark, and model training pipelines lineage events should still work as expected.

Test Strategy/Script

How will this item be verified?

  1. Using create a new aissemble-based project using the latest archetype snapshot.
mvn archetype:generate '-DarchetypeGroupId=com.boozallen.aissemble' \
                           '-DarchetypeArtifactId=foundation-archetype' \
                           '-DarchetypeVersion=1.10.0-SNAPSHOT' \
                           '-DgroupId=org.test' \
                           '-Dpackage=org.test' \
                           '-DprojectGitUrl=test.org/test.git' \
                           '-DprojectName=Test pyspark lineage' \
                           '-DartifactId=test-449' \
    && cd test-449
  1. Set your Java version to 17 if it is not currently

  2. Under -model/src/main/resources/pipelines add below pipeline models SparkPipeline.json, PythonPipeline.json, and ClassificationTraining.json

  3. Fully generate the project by running mvn clean install and following manual actions

  4. Build the project without the cache and follow the last manual action.

    mvn clean install -Dmaven.build.cache.skipCache
  5. Deploy the project and wait for all services ready

    tilt up; tilt down
  6. Manually trigger the python-pipeline pod and verify no errors in the log

  7. Manually trigger the spark-pipeline pod and verify no errors in the log

  8. Use postman or any rest client to trigger the training pipeline and verify a successful training pipeline id responded

References/Additional Context

As needed

@csun-cpointe csun-cpointe added the task Support ticket - typically not code related label Oct 31, 2024
@csun-cpointe csun-cpointe self-assigned this Oct 31, 2024
@csun-cpointe csun-cpointe added the slacktask Small ticket done in down time label Oct 31, 2024
@csun-cpointe csun-cpointe added this to the 1.10.0 milestone Oct 31, 2024
csun-cpointe added a commit that referenced this issue Oct 31, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
csun-cpointe added a commit that referenced this issue Oct 31, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
csun-cpointe added a commit that referenced this issue Oct 31, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
csun-cpointe added a commit that referenced this issue Oct 31, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
csun-cpointe added a commit that referenced this issue Nov 1, 2024
…leanup

#449 data/model lineage legacy namespace cleanup
@carter-cundiff
Copy link
Contributor

Testing passed:
image
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
slacktask Small ticket done in down time task Support ticket - typically not code related
Projects
None yet
Development

No branches or pull requests

2 participants