From 9018bf5a2ac38f14e965dd45af50211262ec8088 Mon Sep 17 00:00:00 2001 From: Dmitry Sorokin Date: Tue, 8 Aug 2023 16:32:09 +0100 Subject: [PATCH 1/2] update docs to reflect change to /conf file structure Signed-off-by: Dmitry Sorokin --- docs/source/data/data_catalog.md | 4 ++-- docs/source/development/commands_reference.md | 4 ++-- docs/source/nodes_and_pipelines/micro_packaging.md | 4 ++-- docs/source/tutorial/add_another_pipeline.md | 8 ++++---- docs/source/tutorial/create_a_pipeline.md | 2 +- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/source/data/data_catalog.md b/docs/source/data/data_catalog.md index 3cc2cbd90b..d7c73e4fdf 100644 --- a/docs/source/data/data_catalog.md +++ b/docs/source/data/data_catalog.md @@ -359,10 +359,10 @@ The list of all available parameters is given in the [Paramiko documentation](ht You can use the [`kedro catalog create` command to create a Data Catalog YAML configuration](../development/commands_reference.md#create-a-data-catalog-yaml-configuration-file). -This creates a `//catalog/.yml` configuration file with `MemoryDataSet` datasets for each dataset in a registered pipeline if it is missing from the `DataCatalog`. +This creates a `//catalog_.yml` configuration file with `MemoryDataSet` datasets for each dataset in a registered pipeline if it is missing from the `DataCatalog`. ```yaml -# //catalog/.yml +# //catalog_.yml rockets: type: MemoryDataSet scooters: diff --git a/docs/source/development/commands_reference.md b/docs/source/development/commands_reference.md index 39c162dafc..45801ea112 100644 --- a/docs/source/development/commands_reference.md +++ b/docs/source/development/commands_reference.md @@ -376,7 +376,7 @@ kedro micropkg pull The above command will take the bundled `.tar.gz` file and do the following: * Place source code in `src//pipelines/` -* Place parameters in `conf/base/parameters/.yml` +* Place parameters in `conf/base/parameters_.yml` * Pull out tests and place in `src/tests/pipelines/` `kedro micropkg pull` works with PyPI, local and cloud storage: @@ -512,7 +512,7 @@ kedro catalog create --pipeline= The command also accepts an optional `--env` argument that allows you to specify a configuration environment (defaults to `base`). -The command creates the following file: `//catalog/.yml` +The command creates the following file: `//catalog_.yml` #### Notebooks diff --git a/docs/source/nodes_and_pipelines/micro_packaging.md b/docs/source/nodes_and_pipelines/micro_packaging.md index a3f9392808..312dfa418e 100644 --- a/docs/source/nodes_and_pipelines/micro_packaging.md +++ b/docs/source/nodes_and_pipelines/micro_packaging.md @@ -35,7 +35,7 @@ When you package your micro-package, such as a modular pipeline for example, Ked Kedro will also include any requirements found in `src//pipelines//requirements.txt` in the micro-package tar file. These requirements will later be taken into account when pulling a micro-package via `kedro micropkg pull`. ```{note} -Kedro will not package the catalog config files even if those are present in `conf//catalog/.yml`. +Kedro will not package the catalog config files even if those are present in `conf//catalog_.yml`. ``` If you plan to publish your packaged micro-package to some Python package repository like [PyPI](https://pypi.org/), you need to make sure that your micro-package name doesn't clash with any of the existing packages in that repository. However, there is no need to rename any of your source files if that is the case. Simply alias your package with a new name by running `kedro micropkg package --alias `. @@ -71,7 +71,7 @@ You can pull a micro-package from a tar file by executing `kedro micropkg pull < * The `` must either be a package name on PyPI or a path to the source distribution file. * Kedro will unpack the tar file, and install the files in following locations in your Kedro project: * All the micro-package code in `src///` - * Configuration files in `conf//parameters/.yml`, where `` defaults to `base`. + * Configuration files in `conf//parameters_.yml`, where `` defaults to `base`. * To place parameters from a different config environment, run `kedro micropkg pull --env ` * Unit tests in `src/tests/` * Kedro will also parse any requirements packaged with the micro-package and add them to project level `requirements.in`. diff --git a/docs/source/tutorial/add_another_pipeline.md b/docs/source/tutorial/add_another_pipeline.md index 3e4c0089e2..95093b5d0b 100644 --- a/docs/source/tutorial/add_another_pipeline.md +++ b/docs/source/tutorial/add_another_pipeline.md @@ -17,7 +17,7 @@ The data science pipeline is made up of the following: * Two python files within `src/spaceflights/pipelines/data_science` * `nodes.py` (for the node functions that form the data processing) * `pipeline.py` (to build the pipeline) -* A yaml file: `conf/base/parameters/data_science.yml` to define the parameters used when running the pipeline +* A yaml file: `conf/base/parameters_data_science.yml` to define the parameters used when running the pipeline * `__init__.py` files in the required folders to ensure that Python can import the pipeline @@ -42,7 +42,7 @@ def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple: Args: data: Data containing features and target. - parameters: Parameters defined in parameters/data_science.yml. + parameters: Parameters defined in parameters_data_science.yml. Returns: Split data. """ @@ -89,7 +89,7 @@ def evaluate_model( ## Input parameter configuration -Parameters that are used by the `DataCatalog` when the pipeline executes are stored in `conf/base/parameters/data_science.yml`: +Parameters that are used by the `DataCatalog` when the pipeline executes are stored in `conf/base/parameters_data_science.yml`:
Click to expand @@ -276,7 +276,7 @@ candidate_modelling_pipeline.regressor: ```

-2. Update the parameters file for the data science pipeline in `conf/base/parameters/data_science.yml` to replace the existing contents for `model_options` with the following for the two instances of the template pipeline: +2. Update the parameters file for the data science pipeline in `conf/base/parameters_data_science.yml` to replace the existing contents for `model_options` with the following for the two instances of the template pipeline:
Click to expand diff --git a/docs/source/tutorial/create_a_pipeline.md b/docs/source/tutorial/create_a_pipeline.md index d0173a1cc9..1828c41929 100644 --- a/docs/source/tutorial/create_a_pipeline.md +++ b/docs/source/tutorial/create_a_pipeline.md @@ -14,7 +14,7 @@ The data processing pipeline prepares the data for model building by combining t * Two python files within `src/spaceflights/pipelines/data_processing` * `nodes.py` (for the node functions that form the data processing) * `pipeline.py` (to build the pipeline) -* A yaml file: `conf/base/parameters/data_processing.yml` to define the parameters used when running the pipeline +* A yaml file: `conf/base/parameters_data_processing.yml` to define the parameters used when running the pipeline * `__init__.py` files in the required folders to ensure that Python can import the pipeline ```{note} From f0f7c8483739b2f319e4262c5ada9b5912645b7d Mon Sep 17 00:00:00 2001 From: Dmitry Sorokin Date: Tue, 8 Aug 2023 18:55:48 +0100 Subject: [PATCH 2/2] update docs, folder structure diagrams Signed-off-by: Dmitry Sorokin --- docs/source/nodes_and_pipelines/micro_packaging.md | 3 +-- docs/source/nodes_and_pipelines/modular_pipelines.md | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/source/nodes_and_pipelines/micro_packaging.md b/docs/source/nodes_and_pipelines/micro_packaging.md index 312dfa418e..3ad1ddcc69 100644 --- a/docs/source/nodes_and_pipelines/micro_packaging.md +++ b/docs/source/nodes_and_pipelines/micro_packaging.md @@ -19,8 +19,7 @@ When you package your micro-package, such as a modular pipeline for example, Ked ```text ├── conf │ └── base -│ └── parameters -│ └── {{pipeline_name*}} <-- All parameter file(s) +│ └── parameters_{{pipeline_name*}} <-- All parameter file(s) └── src ├── my_project │ ├── __init__.py diff --git a/docs/source/nodes_and_pipelines/modular_pipelines.md b/docs/source/nodes_and_pipelines/modular_pipelines.md index 12d69509c6..5064ae7b87 100644 --- a/docs/source/nodes_and_pipelines/modular_pipelines.md +++ b/docs/source/nodes_and_pipelines/modular_pipelines.md @@ -52,8 +52,7 @@ Running the `kedro pipeline create` command adds boilerplate folders and files f ```text ├── conf │ └── base -│ └── parameters -│ └── {{pipeline_name}}.yml <-- Pipeline-specific parameters +│ └── parameters_{{pipeline_name}}.yml <-- Pipeline-specific parameters └── src ├── my_project │ ├── __init__.py