fix the bug for Final table write happens before running final_agg_dq and final_query_dq expectations #22
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phanikumarvemuri
requested review from
asingamaneni and
Umeshsp22
as code owners
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #22 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 22 22
Lines 1369 1369
=========================================
Hits 1369 1369
☔ View full report in Codecov by Sentry. |
Umeshsp22
approved these changes
Sep 6, 2023
|
LGTM |
asingamaneni
added a commit
that referenced
this pull request
Jul 29, 2024
* Merge pull request #1 * Base setup for react ui * Merge pull request #2 * Zustand + React Query setup * Integrating Zustand, RQ and api.github with a basic implementation * Completed the mantine + dev tooling setup and implemented a layout (#3) * Local test, vite, linting and prettier setup. Switching to mantine instead of material ui * Updating readme.md with run scripts details * Adding basic app layout with mantine * Adding auth provider which shows a modal for users to enter their github token (#4) * GitHub Integration + Render Repos List And User Card (#5) * 1. Segregating header into a new component 2. Updating header with new styles * Implement a ReposList component with dropdown. Supports smooth scrolling and autocomplete search ReposList shows in Header component. * Implemented a UserButton Component. UserButton is in Header * Implemented use repos Integrated it with ReposList Implemented useUser Integrated it with UserButton * Update Readme.md * Reverting IDE changes * 1. Improved token handling 2. Improved api-client 3. Added a Loading component * Linting fixes * Failed attempt at wrapping Error and Loading using a HOC. Will investigate this later. * Updates to Header and UserMenu * Add a sample NavBar * updating tests * Implement Base Structure and GitHub OAuth Login (#97) * Initializing theme files (#11) * Adding a basic mantine theme file. It will be updated as the code base develops. Any colors or gaps will not be hardcoded. They should first be defined in the theme and then should be used in the code. * updating .gitignore * Includes a comprehensive test setup with mock data and support structure. (#12) setting up tests in the future will be easier. * 9 tech debt setup public private routing (#13) * Working version of routing * Implemented routing, however tests for the login page needs to figured out. * implemented notification from mantine (#14) * 8 feature implement login functionality using GitHub oauth (#15) * clean up * A simple but beautiful login page * Github Oauth setup * 16 tech debt complete todos (#17) * Fixed testing setup and have all tests for OAuth. * Fixed remaining TODOs, added tests. * Adding a DEVELOPER.md and .env.example * Initializing theme files (#11) * Adding a basic mantine theme file. It will be updated as the code base develops. Any colors or gaps will not be hardcoded. They should first be defined in the theme and then should be used in the code. * updating .gitignore * Includes a comprehensive test setup with mock data and support structure. (#12) setting up tests in the future will be easier. * 9 tech debt setup public private routing (#13) * Working version of routing * Implemented routing, however tests for the login page needs to figured out. * implemented notification from mantine (#14) * 8 feature implement login functionality using GitHub oauth (#15) * clean up * A simple but beautiful login page * Github Oauth setup * 16 tech debt complete todos (#17) * Fixed testing setup and have all tests for OAuth. * Fixed remaining TODOs, added tests. * Adding a DEVELOPER.md and .env.example * Restructuring + layout (#19) * adding yaml file * 20 load list of organizations on the left cell (#21) * WIP * fetch user name before rendering main page * implement api for user repos * show available repos on the left hand side * Rendering an un nested list of files * rendering yaml files in each project * implemented rules table * editing rules.yaml * Build a working version (#22) * working modal * working flow * Updated repos list * Refactored CommitsList.tsx and apis * minor fixes * clean up --------- Co-authored-by: Ashok Singamaneni <ashok.singamaneni@nike.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the issue - #11
Description
when
write_to_tableoption is set, the write should only happen when all dq checks are completed. Right now the table is being saved after running row_dq rules but agg_final or query_final may fail . So moving thewritepart toexpectations.pyRelated Issue
#11
Motivation and Context
If the
agg_finalandquery_finaldq rule fails, there is no point in saving the dataframe as table.How Has This Been Tested?
Ran locally and unit tested
Screenshots (if appropriate):
Types of changes
Checklist: