Refactoring datasets components

[lorenzo-cavazzi]

This is an umbrella issue for refactoring datasets components.

Tasks

Tasks include converting all the datasets components to:

• Use React functional components.
• Invoke APIs through RTK Query.
  • ℹ️ UPDATE: we should phase out the GET /knowledge-graph/datasets API (this: https://github.com/SwissDataScienceCenter/renku-graph/tree/2.28.3/knowledge-graph#get-knowledge-graphdatasets) and use the GET /knowledge-graph/entities to fetch all the datasets data
  • We can be cautious and create a separate API for that, and swap them at the end; if something doesn't work, we don't need to hold on but we should notify KG to add the missing data and then create an issue to swap them in the (near) future
• Save local status to Redux slices using RTK.

Changes

In the process, we should:

• Avoid storing data in the Redux store through the old Model approach, nor storing them locally (remember that even simple preferences stored locally might be reset by ancestor components refreshing at any time since many Datasets components are descendants of old Project components).
• Avoid using the previous From-generator approach; consider using a different library for that (maybe react-final-form, or redux-form if we need to keep the state in Redux for the reasons suggested above), or otherwise just use a very simplified way of preserving transient user inputs in Redux.
• Remove all the datasets components passed as properties from the Project page (in the subComponents function).
• Remember these minor changes from other issues when re-implementing the components
  • Validate dataset keyword input #2356
  • default to not expanding folders in dataset view #1930
  • remove unneeded calls to gitlab graphql on the dataset view page #1912
  • Clicking on a file in the dataset view should show the preview not the lineage #1270

Caveats

Here are a few things to keep in mind.

Datasets can be accessed either through a project or alone. There are some differences:

• As an independent entity they:
  • Have a unique ID, similar to a SHA, that identifies them (there might be new versions, and older versions of these ids might contain - that must be stripped out). There is no user-friendly name. E.G: https://dev.renku.ch/datasets/7492c98815a544418762fcb5fae2d6ba
  • Data is available only from the KG, and it might be temporarily not up-to-date after modifying a dataset. Currently, there is no way around it since the KG takes some time to process new commits.
  • They are usually contained in 1 or more projects; even when contained in just 1 project, they are a separate entity from the project's counterpart.
• Inside a project:
  • Both a unique ID (SHA-style) and a human-friendly ID exist. The readable one works slight differently since it points to the latest version of that dataset. E.G. lorenzo.cavazzi.tech/updated-flight-tutorial-2023/datasets/flight-data might vary when accessing it again later.
  • Data is available from both KG and core. Core is immediately up-to-date but it might have less information. KG might be outdated after updating a dataset. Be careful when merging the data and explore possible good solutions (E.G. play with the unique IDs and see if that can be used to identify outdated data. Maybe we can even discard outdated KG data and try to refetch with polling style until we get the recent information; or we could use the KG indexing status data to discard KG content until the project is fully indexed. In any case, don't mix core and KG data unless it's sure they are both referring to the latest version of the dataset)

There might be other difficulties not listed here. Consider spending some time before the implementation to analyze how datasets work right now and what could be problematic.

[lorenzo-cavazzi]

We can close this. We will continue the refactoring in #2678