-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(datasets): add SparkStreamingDataSet
#198
Commits on May 1, 2023
-
Fix links on GitHub issue templates (kedro-org#150)
Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 46bb394 - Browse repository at this point
Copy the full SHA 46bb394View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for c9421ae - Browse repository at this point
Copy the full SHA c9421aeView commit details -
Migrate most of
kedro-datasets
metadata topyproject.toml
(kedro-……org#161) * Include missing requirements files in sdist Fix kedro-orggh-86. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Migrate most project metadata to `pyproject.toml` See kedro-org/kedro#2334. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Move requirements to `pyproject.toml` Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 63f578a - Browse repository at this point
Copy the full SHA 63f578aView commit details -
restructure the strean dataset to align with the other spark dataset
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 4b387ff - Browse repository at this point
Copy the full SHA 4b387ffView commit details -
adding README.md for specification
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 39ad9fd - Browse repository at this point
Copy the full SHA 39ad9fdView commit details -
Update kedro-datasets/kedro_datasets/spark/spark_stream_dataset.py
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 69eb8be - Browse repository at this point
Copy the full SHA 69eb8beView commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 3106068 - Browse repository at this point
Copy the full SHA 3106068View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for b8141a7 - Browse repository at this point
Copy the full SHA b8141a7View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 738625e - Browse repository at this point
Copy the full SHA 738625eView commit details -
Update kedro-datasets/kedro_datasets/spark/README.md
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for a54cc67 - Browse repository at this point
Copy the full SHA a54cc67View commit details -
add unit tests and SparkStreamingDataset in init.py
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for b924ad6 - Browse repository at this point
Copy the full SHA b924ad6View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 743b823 - Browse repository at this point
Copy the full SHA 743b823View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 3bb3717 - Browse repository at this point
Copy the full SHA 3bb3717View commit details -
Upgrade Polars (kedro-org#171)
* Upgrade Polars Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> * Update Polars to 0.17.x --------- Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for ae3bc87 - Browse repository at this point
Copy the full SHA ae3bc87View commit details -
Configuration menu - View commit details
-
Copy full SHA for eb634a1 - Browse repository at this point
Copy the full SHA eb634a1View commit details -
Migrate
kedro-airflow
to static metadata (kedro-org#172)* Migrate kedro-airflow to static metadata See kedro-org/kedro#2334. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add explicit PEP 518 build requirements for kedro-datasets Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Typos Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Remove dangling reference to requirements.txt Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add release notes Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 115940b - Browse repository at this point
Copy the full SHA 115940bView commit details -
Migrate
kedro-telemetry
to static metadata (kedro-org#174)* Migrate kedro-telemetry to static metadata See kedro-org/kedro#2334. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add release notes Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 35231af - Browse repository at this point
Copy the full SHA 35231afView commit details -
ci: port lint, unit test, and e2e tests to Actions (kedro-org#155)
* Add unit test + lint test on GA * trigger GA - will revert Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Fix lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add end to end tests * Add cache key Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add cache action Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Rename workflow files Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Lint + add comment + default bash Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add windows test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update workflow name + revert changes to READMEs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add kedro-telemetry/RELEASE.md to trufflehog ignore Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add pytables to test_requirements remove from workflow Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert "Add pytables to test_requirements remove from workflow" This reverts commit 8203daa. * Separate pip freeze step Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 8c2ea1b - Browse repository at this point
Copy the full SHA 8c2ea1bView commit details -
Migrate
kedro-docker
to static metadata (kedro-org#173)* Migrate kedro-docker to static metadata See kedro-org/kedro#2334. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Address packaging warning Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix tests Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Actually install current plugin with dependencies Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add release notes Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for a73b216 - Browse repository at this point
Copy the full SHA a73b216View commit details -
Introdcuing .gitpod.yml to kedro-plugins (kedro-org#185)
Currently opening gitpod will installed a Python 3.11 which breaks everything because we don't support it set. This PR introduce a simple .gitpod.yml to get it started. Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 7f4527d - Browse repository at this point
Copy the full SHA 7f4527dView commit details -
sync APIDataSet from kedro's
develop
(kedro-org#184)* Update APIDataSet Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Sync ParquetDataSet Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Sync Test Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Linting Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Revert Unnecessary ParquetDataSet Changes Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Sync release notes Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 57a11d6 - Browse repository at this point
Copy the full SHA 57a11d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 11c3888 - Browse repository at this point
Copy the full SHA 11c3888View commit details -
Configuration menu - View commit details
-
Copy full SHA for 634d884 - Browse repository at this point
Copy the full SHA 634d884View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e8f55c - Browse repository at this point
Copy the full SHA 9e8f55cView commit details -
Configuration menu - View commit details
-
Copy full SHA for dbdf19c - Browse repository at this point
Copy the full SHA dbdf19cView commit details -
Merge remote-tracking branch 'origin/add-stream-datasets' into add-st…
…ream-datasets # Conflicts: # .github/workflows/check-plugin.yml # kedro-datasets/tests/api/test_api_dataset.py
Configuration menu - View commit details
-
Copy full SHA for 4e49fd9 - Browse repository at this point
Copy the full SHA 4e49fd9View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 1a7a477 - Browse repository at this point
Copy the full SHA 1a7a477View commit details -
restructure the strean dataset to align with the other spark dataset
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for e877944 - Browse repository at this point
Copy the full SHA e877944View commit details -
adding README.md for specification
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 09e9cf2 - Browse repository at this point
Copy the full SHA 09e9cf2View commit details -
Update kedro-datasets/kedro_datasets/spark/spark_stream_dataset.py
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 2e30ec0 - Browse repository at this point
Copy the full SHA 2e30ec0View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 6147636 - Browse repository at this point
Copy the full SHA 6147636View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 29376e9 - Browse repository at this point
Copy the full SHA 29376e9View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 42ed37a - Browse repository at this point
Copy the full SHA 42ed37aView commit details -
Update kedro-datasets/kedro_datasets/spark/README.md
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for d93d9b9 - Browse repository at this point
Copy the full SHA d93d9b9View commit details -
add unit tests and SparkStreamingDataset in init.py
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 5b83444 - Browse repository at this point
Copy the full SHA 5b83444View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 5b0630e - Browse repository at this point
Copy the full SHA 5b0630eView commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 1433808 - Browse repository at this point
Copy the full SHA 1433808View commit details -
Configuration menu - View commit details
-
Copy full SHA for c7778b5 - Browse repository at this point
Copy the full SHA c7778b5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7341429 - Browse repository at this point
Copy the full SHA 7341429View commit details -
Configuration menu - View commit details
-
Copy full SHA for d8d3bc2 - Browse repository at this point
Copy the full SHA d8d3bc2View commit details -
Configuration menu - View commit details
-
Copy full SHA for be4a3e5 - Browse repository at this point
Copy the full SHA be4a3e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for d3bc0d2 - Browse repository at this point
Copy the full SHA d3bc0d2View commit details
Commits on May 2, 2023
-
Configuration menu - View commit details
-
Copy full SHA for e39c639 - Browse repository at this point
Copy the full SHA e39c639View commit details -
Configuration menu - View commit details
-
Copy full SHA for 66440f4 - Browse repository at this point
Copy the full SHA 66440f4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ed5b90 - Browse repository at this point
Copy the full SHA 0ed5b90View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 04c623b - Browse repository at this point
Copy the full SHA 04c623bView commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for a76f944 - Browse repository at this point
Copy the full SHA a76f944View commit details -
remove code snippets fpr testing
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 30b002d - Browse repository at this point
Copy the full SHA 30b002dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9bef3a2 - Browse repository at this point
Copy the full SHA 9bef3a2View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 0bb5fe1 - Browse repository at this point
Copy the full SHA 0bb5fe1View commit details
Commits on May 4, 2023
-
update test and remove redundacy
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for e0ebe27 - Browse repository at this point
Copy the full SHA e0ebe27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5bb5766 - Browse repository at this point
Copy the full SHA 5bb5766View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 2075781 - Browse repository at this point
Copy the full SHA 2075781View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for e8ea0d3 - Browse repository at this point
Copy the full SHA e8ea0d3View commit details -
docs: Add community contributions (kedro-org#199)
* Add community contributions Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Use newer link to docs Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for f08dd09 - Browse repository at this point
Copy the full SHA f08dd09View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 24bb527 - Browse repository at this point
Copy the full SHA 24bb527View commit details -
update test and remove redundacy
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 437e77e - Browse repository at this point
Copy the full SHA 437e77eView commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for a3fdbf6 - Browse repository at this point
Copy the full SHA a3fdbf6View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 9d60f25 - Browse repository at this point
Copy the full SHA 9d60f25View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for ced007d - Browse repository at this point
Copy the full SHA ced007dView commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 0b88324 - Browse repository at this point
Copy the full SHA 0b88324View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for ed26aad - Browse repository at this point
Copy the full SHA ed26aadView commit details -
Configuration menu - View commit details
-
Copy full SHA for 170b092 - Browse repository at this point
Copy the full SHA 170b092View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for e63a53a - Browse repository at this point
Copy the full SHA e63a53aView commit details -
Configuration menu - View commit details
-
Copy full SHA for d986c75 - Browse repository at this point
Copy the full SHA d986c75View commit details -
Configuration menu - View commit details
-
Copy full SHA for 88e6ee4 - Browse repository at this point
Copy the full SHA 88e6ee4View commit details
Commits on May 5, 2023
-
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 64232fa - Browse repository at this point
Copy the full SHA 64232faView commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 8a61b41 - Browse repository at this point
Copy the full SHA 8a61b41View commit details
Commits on May 16, 2023
-
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 37e66e8 - Browse repository at this point
Copy the full SHA 37e66e8View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 07032a8 - Browse repository at this point
Copy the full SHA 07032a8View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 2470de1 - Browse repository at this point
Copy the full SHA 2470de1View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for c4e0f4e - Browse repository at this point
Copy the full SHA c4e0f4eView commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 7e3555e - Browse repository at this point
Copy the full SHA 7e3555eView commit details
Commits on May 17, 2023
-
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 6a0029d - Browse repository at this point
Copy the full SHA 6a0029dView commit details
Commits on May 23, 2023
-
fix streaming dataset configurations
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for e8f6696 - Browse repository at this point
Copy the full SHA e8f6696View commit details
Commits on May 25, 2023
-
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 9a5ebad - Browse repository at this point
Copy the full SHA 9a5ebadView commit details -
resolve comments re documentation
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for eacdd46 - Browse repository at this point
Copy the full SHA eacdd46View commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 68b6e1b - Browse repository at this point
Copy the full SHA 68b6e1bView commit details -
Signed-off-by: Tingting_Wan <tingting_wan@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 5b2a479 - Browse repository at this point
Copy the full SHA 5b2a479View commit details
Commits on May 26, 2023
-
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Configuration menu - View commit details
-
Copy full SHA for b94f211 - Browse repository at this point
Copy the full SHA b94f211View commit details
Commits on May 30, 2023
-
test(docker): remove outdated logging-related step (kedro-org#207)
* fixkedro- docker e2e test Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * fix: add timeout to request to satisfy bandit lint --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 9381816 - Browse repository at this point
Copy the full SHA 9381816View commit details -
ci: ensure plugin requirements get installed in CI (kedro-org#208)
* ci: install the plugin alongside test requirements * ci: install the plugin alongside test requirements * Update kedro-airflow.yml * Update kedro-datasets.yml * Update kedro-docker.yml * Update kedro-telemetry.yml * Update kedro-airflow.yml * Update kedro-datasets.yml * Update kedro-airflow.yml * Update kedro-docker.yml * Update kedro-telemetry.yml * ci(telemetry): update isort config to correct sort * Don't use profile ¯\_(ツ)_/¯ Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore(datasets): remove empty `tool.black` section * chore(docker): remove empty `tool.black` section --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 373e166 - Browse repository at this point
Copy the full SHA 373e166View commit details -
ci: Migrate the release workflow from CircleCI to GitHub Actions (ked…
…ro-org#203) * Create check-release.yml * change from test pypi to pypi * split into jobs and move version logic into script * update github actions output * lint * changes based on review * changes based on review * fix script to not append continuously * change pypi api token logic Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for f033b95 - Browse repository at this point
Copy the full SHA f033b95View commit details -
build: Relax Kedro bound for
kedro-datasets
(kedro-org#140)* Less strict pin on Kedro for datasets Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 3fdb71c - Browse repository at this point
Copy the full SHA 3fdb71cView commit details -
ci: don't run checks on both
push
/pull_request
(kedro-org#192)* ci: don't run checks on both `push`/`pull_request` * ci: don't run checks on both `push`/`pull_request` * ci: don't run checks on both `push`/`pull_request` * ci: don't run checks on both `push`/`pull_request` Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for b08aa6f - Browse repository at this point
Copy the full SHA b08aa6fView commit details -
chore: delete extra space ending check-release.yml (kedro-org#210)
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 148b464 - Browse repository at this point
Copy the full SHA 148b464View commit details -
ci: Create merge-gatekeeper.yml to make sure PR only merged when all …
…tests checked. (kedro-org#215) * Create merge-gatekeeper.yml * Update .github/workflows/merge-gatekeeper.yml --------- Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for be2431c - Browse repository at this point
Copy the full SHA be2431cView commit details -
ci: Remove the CircleCI setup (kedro-org#209)
* remove circleci setup files and utils * remove circleci configs in kedro-telemetry * remove redundant .github in kedro-telemetry * Delete continue_config.yml * Update check-release.yml * lint * increase timeout to 40 mins for docker e2e tests Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 74a211f - Browse repository at this point
Copy the full SHA 74a211fView commit details -
feat: Dataset API add
save
method (kedro-org#180)* [FEAT] add save method to APIDataset Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] create save_args parameter for api_dataset Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] add tests for socket + http errors Signed-off-by: <jmcdonnell@fieldbox.ai> Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] check save data is json Signed-off-by: <jmcdonnell@fieldbox.ai> Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] clean code Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] handle different data types Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] test coverage for exceptions Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] add examples in APIDataSet docstring Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * sync APIDataSet from kedro's `develop` (kedro-org#184) * Update APIDataSet Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Sync ParquetDataSet Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Sync Test Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Linting Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Revert Unnecessary ParquetDataSet Changes Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Sync release notes Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] remove support for delete method Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] lint files Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] fix conflicts Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] remove fail save test Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] review suggestions Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [ENH] fix tests Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> * [FIX] reorder arguments Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> --------- Signed-off-by: jmcdonnell <jmcdonnell@fieldbox.ai> Signed-off-by: <jmcdonnell@fieldbox.ai> Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Co-authored-by: jmcdonnell <jmcdonnell@fieldbox.ai> Co-authored-by: Nok Lam Chan <mediumnok@gmail.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 9d7820a - Browse repository at this point
Copy the full SHA 9d7820aView commit details -
ci: Automatically extract release notes for GitHub Releases (kedro-or…
…g#212) * ci: Automatically extract release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * fix lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Raise exceptions Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Lint Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 36de4b9 - Browse repository at this point
Copy the full SHA 36de4b9View commit details -
feat: Add metadata attribute to datasets (kedro-org#189)
* Add metadata attribute to all datasets Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 870e623 - Browse repository at this point
Copy the full SHA 870e623View commit details -
feat: Add ManagedTableDataset for managed Delta Lake tables in Databr…
…icks (kedro-org#206) * committing first version of UnityTableCatalog with unit tests. This datasets allows users to interface with Unity catalog tables in Databricks to both read and write. Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * renaming dataset Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * adding mlflow connectors Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * fixing mlflow imports Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * cleaned up mlflow for initial release Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * cleaned up mlflow references from setup.py for initial release Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * fixed deps in setup.py Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * adding comments before intiial PR Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * moved validation to dataclass Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * bug fix in type of partition column and cleanup Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * updated docstring for ManagedTableDataSet Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * added backticks to catalog Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * fixing regex to allow hyphens Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/test_requirements.txt Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * adding backticks to catalog Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Require pandas < 2.0 for compatibility with spark < 3.4 Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Replace use of walrus operator Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add test coverage for validation methods Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove unused versioning functions Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Fix exception catching for invalid schema, add test for invalid schema Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add pylint ignore Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add tests/databricks to ignore for no-spark tests Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Nok Lam Chan <mediumnok@gmail.com> * Update kedro-datasets/kedro_datasets/databricks/managed_table_dataset.py Co-authored-by: Nok Lam Chan <mediumnok@gmail.com> * Remove spurious mlflow test dependency Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add explicit check for database existence Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove character limit for table names Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Refactor validation steps in ManagedTable Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove spurious checks for table and schema name existence Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> --------- Signed-off-by: Danny Farah <danny_farah@mckinsey.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Danny Farah <danny.farah@quantumblack.com> Co-authored-by: Danny Farah <danny_farah@mckinsey.com> Co-authored-by: Nok Lam Chan <mediumnok@gmail.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 9d66cc8 - Browse repository at this point
Copy the full SHA 9d66cc8View commit details -
docs: Update APIDataset docs and refactor (kedro-org#217)
* Update APIDataset docs and refactor * Acknowledge community contributor * Fix more broken doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Lint Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix release notes of upcoming kedro-datasets --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 0aaa922 - Browse repository at this point
Copy the full SHA 0aaa922View commit details -
feat: Release
kedro-datasets
version1.3.0
(kedro-org#219)* Modify release version and RELEASE.md Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add proper name for ManagedTableDataSet Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update kedro-datasets/RELEASE.md Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Revert lost semicolon for release 1.2.0 Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> --------- Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for ccec03b - Browse repository at this point
Copy the full SHA ccec03bView commit details -
docs: Fix APIDataSet docstring (kedro-org#220)
* Fix APIDataSet docstring Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add release notes Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Separate [docs] extras from [all] in kedro-datasets Fix kedro-orggh-143. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for c2a7128 - Browse repository at this point
Copy the full SHA c2a7128View commit details -
Update kedro-datasets/tests/spark/test_spark_streaming_dataset.py
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 64446dc - Browse repository at this point
Copy the full SHA 64446dcView commit details -
Update kedro-datasets/kedro_datasets/spark/spark_streaming_dataset.py
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 497001d - Browse repository at this point
Copy the full SHA 497001dView commit details -
Update kedro-datasets/setup.py
Co-authored-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for 7f25f3c - Browse repository at this point
Copy the full SHA 7f25f3cView commit details -
Configuration menu - View commit details
-
Copy full SHA for bd88b99 - Browse repository at this point
Copy the full SHA bd88b99View commit details -
Signed-off-by: Tom Kurian <tom_kurian@mckinsey.com>
Configuration menu - View commit details
-
Copy full SHA for c094db1 - Browse repository at this point
Copy the full SHA c094db1View commit details