Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: link data connectors to projects #410

Merged

Conversation

leafty
Copy link
Member

@leafty leafty commented Sep 17, 2024

Merging into leafty/build-data-connectors (feature branch).

Adds the possibility to link a data connector to a project.

Details:

  • Add API to link a data connector to a project
  • Add API to list projects connected to a data connector
  • Add API to list data connectors connected to a project

Note that read permissions are cascaded to direct members of the connected project.

  • Creating a link from a private data connector to a project requires the editor role on both sides of the link.
  • Creating a link from a public data connector to a project requires the editor role on the project.
  • Removing a link requires either the editor role on the project or the owner role on the data connector.

/deploy

Moves the `project_slugs` table to `entity_slugs` and `project_slugs_old` to `entity_slugs_old`. This is done to prepare for more entity types to be namespaced.

Note: the sequence migration has been tested:
1. Force migration revision to be `9058bf0a1a12` (before this).
2. Deploy with image tag `renku/renku-data-service:sha-345a1bb` (current `main`) and run `alembic check` and `alembic heads` to verify the DB is before this change.
3. Create some projects.
4. Re-deploy this PR and check that migration revision is `a11752a5afba` (after this).
5. Create more projects.
Introduce namespaced data connectors for Renku 2.0.

Details:
* Remove API routes for `/storages_v2`
* Add API routes for `/data_connectors`
* Add `data_connectors` table
* Drop the `storage_secrets` table
* Add `data_connectors_secrets` table
@leafty leafty force-pushed the leafty/feat-namespaced-data-connectors-4 branch from 44e1704 to e06d710 Compare September 17, 2024 13:20
@RenkuBot
Copy link
Contributor

You can access the deployment of this PR at https://renku-ci-ds-410.dev.renku.ch

Fixes deleting storage configuration fields by setting them to `null`.
…to leafty/feat-namespaced-data-connectors-3

# Conflicts:
#	test/bases/renku_data_services/data_api/test_data_connectors.py
…ectors-3' into leafty/feat-namespaced-data-connectors-4
leafty and others added 4 commits September 20, 2024 16:30
Co-authored-by: Tasko Olevski <olevski90@gmail.com>
…to leafty/feat-namespaced-data-connectors-3

# Conflicts:
#	components/renku_data_services/app_config/config.py
#	components/renku_data_services/data_connectors/api.spec.yaml
#	components/renku_data_services/data_connectors/apispec.py
#	components/renku_data_services/data_connectors/blueprints.py
#	components/renku_data_services/data_connectors/db.py
#	components/renku_data_services/data_connectors/models.py
#	test/bases/renku_data_services/data_api/test_data_connectors.py
Base automatically changed from leafty/feat-namespaced-data-connectors-3 to leafty/build-data-connectors September 23, 2024 12:39
…to leafty/feat-namespaced-data-connectors-4

# Conflicts:
#	components/renku_data_services/authz/authz.py
#	components/renku_data_services/data_connectors/apispec.py
#	components/renku_data_services/data_connectors/db.py
#	components/renku_data_services/data_connectors/models.py
@leafty leafty marked this pull request as ready for review September 24, 2024 11:12
@leafty leafty requested a review from a team as a code owner September 24, 2024 11:12
@m-alisafaee m-alisafaee self-assigned this Sep 27, 2024
Copy link
Contributor

@m-alisafaee m-alisafaee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Flora! Please see my comments.

components/renku_data_services/authz/authz.py Outdated Show resolved Hide resolved
components/renku_data_services/data_connectors/db.py Outdated Show resolved Hide resolved
).one_or_none()
if project is None:
raise errors.MissingResourceError(
message=f"Project with id '{link.project_id}' does not exist or you do not have access to it."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

components/renku_data_services/data_connectors/db.py Outdated Show resolved Hide resolved
@leafty leafty merged commit b3e1e51 into leafty/build-data-connectors Sep 30, 2024
19 of 20 checks passed
@leafty leafty deleted the leafty/feat-namespaced-data-connectors-4 branch September 30, 2024 13:35
@RenkuBot
Copy link
Contributor

Tearing down the temporary RenkuLab deplyoment for this PR.

leafty added a commit that referenced this pull request Sep 30, 2024
Adds the possibility to link a data connector to a project.

Details:
* Add API to link a data connector to a project
* Add API to list projects connected to a data connector
* Add API to list data connectors connected to a project

Note that `read` permissions are cascaded to direct members of the connected project.

* Creating a link from a private data connector to a project requires the `editor` role on both sides of the link.
* Creating a link from a public data connector to a project requires the `editor` role on the project.
* Removing a link requires either the `editor` role on the project or the `owner` role on the data connector.
leafty added a commit that referenced this pull request Sep 30, 2024
Adds the possibility to link a data connector to a project.

Details:
* Add API to link a data connector to a project
* Add API to list projects connected to a data connector
* Add API to list data connectors connected to a project

Note that `read` permissions are cascaded to direct members of the connected project.

* Creating a link from a private data connector to a project requires the `editor` role on both sides of the link.
* Creating a link from a public data connector to a project requires the `editor` role on the project.
* Removing a link requires either the `editor` role on the project or the `owner` role on the data connector.
leafty added a commit that referenced this pull request Oct 4, 2024
Adds the possibility to link a data connector to a project.

Details:
* Add API to link a data connector to a project
* Add API to list projects connected to a data connector
* Add API to list data connectors connected to a project

Note that `read` permissions are cascaded to direct members of the connected project.

* Creating a link from a private data connector to a project requires the `editor` role on both sides of the link.
* Creating a link from a public data connector to a project requires the `editor` role on the project.
* Removing a link requires either the `editor` role on the project or the `owner` role on the data connector.
leafty added a commit that referenced this pull request Oct 17, 2024
Feature branch to add support for data connectors.

See: SwissDataScienceCenter/renku#3751.

Changes:
1. #384
2. #386
3. #406
4. #410
5. #413
6. #443
7. #422
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants