feat(datasets): user can import datasets though the UI

[vfried]

Closes #528

There is a new button in the UI so the user can import datasets using the UI.

There are some things to be fixed in the backend for this to properly work.

Button:

Form:

Deployment: https://virginiatest.dev.renku.ch/

[rokroskar]

preview?

[vfried]

[vfried]

Updated... and ready to be tested

Deployment: https://virginiatest.dev.renku.ch/

[rokroskar]

What is the URL that this lives on? What I mean is, can I do something like

https://renku/projects/myproject/datasets/import?url=doi://12345

This would make integration with third party pages like dataverse datasets much easier I guess?

[vfried]

What is the URL that this lives on? What I mean is, can I do something like

https://renku/projects/myproject/datasets/import?url=doi://12345

This would make integration with third party pages like dataverse datasets much easier I guess?

At the moment you can't do that but we can make a new issue to allow this to happen, I guess if you where to put this url in the browser you would be redirected to renku, the field would have the doi and you need to press the import button....

[rokroskar]

Yes! we should investigate what the best flow would be for this, but it would be super useful.

[vfried]

Yes! we should investigate what the best flow would be for this, but it would be super useful.

I edited my comment on top...

[ciyer]

This is a nice feature. We should use it in the tutorial. I like the UI and the code looks great as well.

I have a few bugs that came up in testing:

1. "Enter" in the URI/DOI field causes the page to reload; I would expect it to submit the form

2. Put junk into the URI/DOI field, and you get "Please wait, dataset import will start soon" forever

3. If you try to import the same file twice, the server returns with an error, but the UI stays in "Please wait, dataset import will start soon"

[vfried]

Fixed:

1. The enter was actually a problem on how the form was built, the button had the onClick event instead of the Form having the onSubmit event.

2. This is a backend problem, i also detected this but i forgot to say to sam. I actually stop the dataset import after a looong time of waiting (the problem is i don't want to just tell the user that dataset import failed too fast)

3. I also detected this problem and forgot to tell to sam, i am not getting anything in the UI that makes me realise that there is an error... all i get is that the job was ENQUEUED.

[jsam]

I added tests for those cases and things seem to pass there correctly now.

2. (In tests) This case splits into 2 possible outcomes:
   2.1) requested format is invalid in which case you should receive the following response:

{
   "result":{
      "updated_at":"2020-03-05T17:27:12.431676",
      "extras":{
         "error":"Invalid parameter value - Could not process junkjunkjunk.\nCannot parse URL."
      },
      "job_id":"f938c53eac7b43ed8b2462146cbfa7dd",
      "created_at":"2020-03-05T17:27:12.309677",
      "state":"FAILED"
   }
}

2.2) requested uri could not be resolved (for example invalid doi or url) in which case you should receive the following response:

{
   "result":{
      "created_at":"2020-03-05T17:32:12.233638",
      "updated_at":"2020-03-05T17:32:12.783466",
      "extras":{
         "error":"Invalid parameter value - Could not process 10.5281/zenodo.11111111111111111.\nURI not found."
      },
      "state":"FAILED",
      "job_id":"387d7e7970024b048be0fc272156f888"
   }
}

3. In this case you should receive response like this:

{
   "result":{
      "created_at":"2020-03-05T17:18:11.089451",
      "updated_at":"2020-03-05T17:18:30.178826",
      "extras":{
         "description":"pybossa-v3.1.2.zip",
         "total_size":714923,
         "progress_size":714923,
         "error":"Dataset exists: \"scifabricpybossa_v31_v312\"."
      },
      "job_id":"1cfc032c7cdd4791881cb8ae1c14ddcb",
      "state":"FAILED"
   }
}

Those error message are coming directly from the core module so they should match the same behavior as we have on the CLI. If you need messaging to be improved we should improve it there.

Let me know if you are not receiving expected results (after we merge the PR).

[vfried]

@jsam thanks for the changes, i will test them ASAP!!!

[lorenzo-cavazzi]

Nice addition!

I have found a couple of minor issues in the code. Currently, there is an error in the src/project/datasets/DatasetsListView.js file preventing me to open the dataset page in a project

[vfried]

@lorenzo-cavazzi i just fixed the lint problems and also the error when entering dataset pages, i think this was related to some other PR, should be fixed now after the rebasing.

[lorenzo-cavazzi]

Great feature! It works well 👍
I have found a couple of minor details (a missing space in text and <Col> sizing) to be adjusted.

I think the DatasetImport.present component has part of the code that should go to the DatasetImport.container since it's about getting data from the backend and not about the presentation of the content. Feel free not to address this if you think it's not needed, but it may help to maintain the componet in the future.

[lorenzo-cavazzi]

All good! I am sure this feature will be appreciated by the users