feat: use cloud storage from amalthea

[olevski]

Adds cloud storage for the new amalthea operator and with the new csi rclone version.

/deploy renku=feat-jupyter-free-sessions amalthea-sessions=feat-add-cloud-storage renku-notebooks=feat-jupyter-free-sessions renku-gateway=feat-add-renku-refresh-token-for-data-service renku-ui=andrea/custom-launcher-jupyter-free extra-values=amalthea-sessions.deployCrd=false,amalthea-sessions.rcloneStorageClass=csi-rclone-387-secret-annotation #notest

[olevski]

This change is part of the following stack:

• feat: add new apispec for the new amalthea sessions #360
  • feat!: expand environments specification #338
    • feat: run notebooks in data service #375
      • feat: use cloud storage from amalthea #387 ◀
        • feat: perf improvements for amalthea sessions #411

_{Change managed by git-spice.}

[RenkuBot]

You can access the deployment of this PR at https://renku-ci-ds-387.dev.renku.ch

[Panaetius]

General Question: there's still #347 that does a bit of cleanup in cloud storage and probably will cause quite a bit of conflict with this, so I'm wondering what the best way would be for us to bring this together with your changes. There's also lots of changes @leafty is working on in regards to cloud storage, and 3 people working on the same bits of code at the same time is bound to cause issues. maybe we should sync?

[Panaetius]

question: historically, users could mount arbitrary, unsaved cloud storage and the UI would just post a body here, in which case it wouldn't have a storage_id. I don't think this is around in v2 but v1 might still use this. So I'm not entirely sure if this if statement will break things and we'd have to generate ad-hoc ids or if this is fine?

[olevski]

For this first implementation of the cloud storage in V2 lets keep things simple and not allow people to mount cloud storage that is unsaved. Then if needed we can add this functionality. This also needs some design because in theory we could allow this in many other places in the API where we currently expect something that exists in the DB. So I will make the storage ID required in the post request to launch a session. If a payload is provided then it just overrides the cloud storage saved in our DB.

[olevski]

General Question: there's still #347 that does a bit of cleanup in cloud storage and probably will cause quite a bit of conflict with this, so I'm wondering what the best way would be for us to bring this together with your changes. There's also lots of changes @leafty is working on in regards to cloud storage, and 3 people working on the same bits of code at the same time is bound to cause issues. maybe we should sync?

When it comes to #347 I think we should split out the API breaking change and merge that at some other time. There is still a lot of testing and minor fixes that are needed on the UI as well. So if we can not have the API breaking change then we dont have to distract the UI with these changes (in addition to many already introduced with the new amalthea). You opened #347 @Panaetius do you think you can split out the API breaking change?

As for the cloud storage changes that @leafty will introduce. I think she is standing by until we merge more things. So after we get this PR and the one furtehr in the stack (#411) I think that @leafty can start targeting the same release branch we have been targeting.

But yeah just to be safe lets have a quick chat in the afternoon tomorrow.

[leafty]

As for the cloud storage changes that @leafty will introduce. I think she is standing by until we merge more things. So after we get this PR and the one furtehr in the stack (#411) I think that @leafty can start targeting the same release branch we have been targeting.

If I understand things correctly, the new sessions API will not support Renku 1.0. In which case the changes should be reasonable.

[leafty]

Note: waiting for a rebase, so I can try to work from this branch.

[Panaetius]

When it comes to #347 I think we should split out the API breaking change and merge that at some other time.

but the API is currently broken and that it works is mostly an accident, any update to sanic or one of its components could break it. So yes, it's a breaking API change but at the same time it's more of a bugfix and unbreaking the API. I wouldn't hold off on this. And it is needed for that PR. Also SwissDataScienceCenter/renku-ui#3299 addressed those changes on the UI AND addresses the changes to storage, so there's a bit of a triangle dependency here.

[RenkuBot]

Tearing down the temporary RenkuLab deplyoment for this PR.