fix(datasets): Correct pyproject.toml syntax for optional dependencies

[DimedS]

Description

#263 modified optional-dependencies in pyproject.toml, leading to issues with dependency loading for spark when running kedro-datasets[spark.SparkDataSet, pandas.ParquetDataSet] (as seen in the iris-databricks starter). This PR corrects the pyproject.toml syntax to resolve the issue.

Development notes

Tested manually using the iris-databricks starter only.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

I confirm that this does not work:

$ pip install "kedro-datasets[spark.SparkDataSet,pandas.ParquetDataSet] @ git+https://github.com/kedro-org/kedro-plugins@main#subdirectory=kedro-datasets"
Looking in indexes: https://juan_luis_cano%40mckinsey.com:****@mckinsey.jfrog.io/artifactory/api/pypi/python/simple
Collecting kedro-datasets[pandas.ParquetDataSet,spark.SparkDataSet]@ git+https://github.com/kedro-org/kedro-plugins@main#subdirectory=kedro-datasets
  Cloning https://github.com/kedro-org/kedro-plugins (to revision main) to /private/var/folders/r7/ywj0_kvj0mxfkdkx0jrgh73r0000gn/T/pip-install-s7t_n1q6/kedro-datasets_26106a3c749d43b88ad77b5a358193e0
  Running command git clone --filter=blob:none --quiet https://github.com/kedro-org/kedro-plugins /private/var/folders/r7/ywj0_kvj0mxfkdkx0jrgh73r0000gn/T/pip-install-s7t_n1q6/kedro-datasets_26106a3c749d43b88ad77b5a358193e0
  Resolved https://github.com/kedro-org/kedro-plugins to commit ce957f1b4d9b776a8389843cee4d7fb31afdaaec
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
WARNING: kedro-datasets 1.5.1 does not provide the extra 'pandas.parquetdataset'
WARNING: kedro-datasets 1.5.1 does not provide the extra 'spark.sparkdataset'

and this does work:

$ pip install "kedro-datasets[spark.SparkDataSet,pandas.ParquetDataSet] @ git+https://github.com/kedro-org/kedro-plugins@fix/290-sparkdataset-dependencies#subdirectory=kedro-datasets"
...
Successfully installed docopt-0.6.2 hdfs-2.7.2 py4j-0.10.9.7 pyspark-3.4.1 s3fs-0.4.2

[merelcht]

LGTM! 👍

[noklam]

Looks good to me in general. I want to do a manual test and re-order some of the dependencies to reflect the structures. I will commit it directly since git doesn't allow to comment on unchanged code.

[noklam]

Tested with pip install .[spark.SparkDataSet] and it works as expected.

I made a few changes:

1. reorder some dependencies to reflect the alphabetical order
2. deleted polars-base and delta-base since this is inconsistent, they are only used by 1 dataset. If we need base for each library we should have pillow-base etc, I don't think we should include this unless necessary.
3. I move all base to the top of the optional-dependencies to reflect the structure.

The use of xxxx-base is not very consistent in current settings. i.e. polars-base is only used in one dataset, I don't think we should keep this.

[deepyaman]

Looks great!

I assume we don't have to count this as a breaking change? Some people may have started using the - delimiter, but I guess they can switch back, since it was never intended.

[noklam]

@deepyaman IMO this is not a breaking change, but rather a bug fix since we never intend to release the spark-dataset syntax. We didn't avoid this problem completely because kedro-datasets[s3-base] still exist, but I don't see better workaround.

[astrojuanlu]

I also see this as a bug fix, these names were working before