Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge data connectors and notebooks/amalthea migration #428

Conversation

leafty
Copy link
Member

@leafty leafty commented Sep 26, 2024

WIP

Merges:

Currently working:

  • Can create a project with the UI or swagger
  • Can create a data connector with the UI or swagger
  • Can link the two with swagger
  • Can create a session launcher with swagger
  • Can launch the session with swagger
  • Public access data is mounted

/deploy renku=feat-jupyter-free-sessions amalthea-sessions=feat-add-cloud-storage renku-notebooks=master renku-gateway=feat-add-renku-refresh-token-for-data-service renku-ui=build/data-connectors extra-values=amalthea-sessions.deployCrd=false,amalthea-sessions.rcloneStorageClass=csi-rclone-387-secret-annotation #notest

Moves the `project_slugs` table to `entity_slugs` and `project_slugs_old` to `entity_slugs_old`. This is done to prepare for more entity types to be namespaced.

Note: the sequence migration has been tested:
1. Force migration revision to be `9058bf0a1a12` (before this).
2. Deploy with image tag `renku/renku-data-service:sha-345a1bb` (current `main`) and run `alembic check` and `alembic heads` to verify the DB is before this change.
3. Create some projects.
4. Re-deploy this PR and check that migration revision is `a11752a5afba` (after this).
5. Create more projects.
Introduce namespaced data connectors for Renku 2.0.

Details:
* Remove API routes for `/storages_v2`
* Add API routes for `/data_connectors`
* Add `data_connectors` table
* Drop the `storage_secrets` table
* Add `data_connectors_secrets` table
Fixes deleting storage configuration fields by setting them to `null`.
…to leafty/feat-namespaced-data-connectors-3

# Conflicts:
#	test/bases/renku_data_services/data_api/test_data_connectors.py
…ectors-3' into leafty/feat-namespaced-data-connectors-4
…ectors-4' into leafty/feat-namespaced-data-connectors-5
…ectors-4' into leafty/feat-namespaced-data-connectors-5
@RenkuBot
Copy link
Contributor

You can access the deployment of this PR at https://renku-ci-ds-428.dev.renku.ch

@olevski olevski force-pushed the feat-add-cloudstorage branch 2 times, most recently from bf8efeb to 04553ec Compare September 27, 2024 13:03
Co-authored-by: Samuel Gaist <samuel.gaist@idiap.ch>
Base automatically changed from feat-add-cloudstorage to release-amaltheas-migration September 27, 2024 14:01
…o leafty/feat-namespaced-data-connectors-merge-amalthea-migration

# Conflicts:
#	Makefile
#	bases/renku_data_services/data_api/app.py
#	components/renku_data_services/notebooks/api.spec.yaml
#	components/renku_data_services/notebooks/api/classes/k8s_client.py
#	components/renku_data_services/notebooks/api/classes/server.py
#	components/renku_data_services/notebooks/api/classes/server_manifest.py
#	components/renku_data_services/notebooks/api/schemas/servers_get.py
#	components/renku_data_services/notebooks/apispec.py
#	components/renku_data_services/notebooks/blueprints.py
#	components/renku_data_services/session/api.spec.yaml
#	components/renku_data_services/storage/blueprints.py
#	components/renku_data_services/storage/models.py
#	poetry.lock
#	projects/renku_data_service/pyproject.toml
#	projects/secrets_storage/pyproject.toml
#	test/bases/renku_data_services/data_api/test_sessions.py
…o leafty/feat-namespaced-data-connectors-merge-amalthea-migration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not merge A PR that has been approved but should not be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants