Getting value from environment variables for maxBufferingDurationMs. … #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Information about it added to README
ilya-kozyrev
approved these changes
Mar 22, 2021
RaphaelSanamyan
pushed a commit
that referenced
this pull request
Jun 7, 2021
… to tokenize sensitive data * [WIP] Transfer from DataflowTemplates to Beam * Move to beam repo * moved convertors for GCSio * Renaming + readme * build errors * minimize suppress * remove UDF usages * Fixes for stylechecks * grooming for data protectors * grooming for data protectors * fix javadoc * Added support for window writing; Fixed ordering in tokenization process * supressed checkstyle errors for BigTableIO class * add data tokenization tests * Changed GCS to FileSystem and removed redundant function from SchemasUtils * Updated README.md for local run with BigQuery sink * add docstring * Updated README.md * Updated README.md and added javadoc for the main pipeline class * remove unused test case * Style fix * Whitespaces fix * Fixed undeclared dependencies and excluded .csv resource files from license analysis * Fix for incorrect rpc url * Fix for nullable types * Data tokenization example group into batches (#11) * GroupIntoBatches was used in the data tokenization pipeline * io files were renamed for the data tokenization template * code format fixed * Getting value from environment variables for maxBufferingDurationMs. Information about it added to README (#14) Getting value from environment variables for maxBufferingDurationMs * [DATAFLOW-139] Incorrect DSG url lead to NPE (#13) * Fix bug incorrect DSG url lead to NPE DATAFLOW-139 Co-authored-by: daria-malkova <daria.malkova@akvelon.com> Co-authored-by: Ilya Kozyrev <ilya.kozyrev@akvelon.com> Co-authored-by: Ramazan Yapparov <ramazan.yapparov@akvelon.com> Co-authored-by: Mikhail Medvedev <mikhail.medvedev@akvelon.com> Co-authored-by: Nuzhdina-Elena <79855159+Nuzhdina-Elena@users.noreply.github.com> Co-authored-by: Nuzhdina-Elena <elena.nuzhdina@akvelon.com> Co-authored-by: MikhailMedvedevAkvelon <78736905+MikhailMedvedevAkvelon@users.noreply.github.com>
RaphaelSanamyan
pushed a commit
that referenced
this pull request
Jun 7, 2021
…at streams data from twitter * # This is a combination of 2 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 2 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #2: # This is a combination of 3 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #3: # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 3 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e44 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #3: # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix Final Commit with all changes Added unit test adding examples for usage usage for TwitterIO added and Java PreCommit failure fix Spotless PreCommit failure fix * Unit test for multiple config added, and beautification * Spotless apply fixed * Removing redundant comments * Removing newly added test * adding newly added test back
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Getting value for max duration of buffering rows from environment variables.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.