User Flow for the Upload Files section in GCBM Simulation Editor

[YashKandalkar]

According to the new changes on the backend, all the configs which are dependent on the .tiff files (along with some other files) are auto-generated.

I wanted to discuss the flow of the user on the front-end for uploading the files, reviewing the config generated on the backend, making any changes if necessary, and finally running the simulation.

I was thinking we can keep 4 pages for uploading files - Disturbances, Classifiers, Input Database, and Miscellaneous (as shown in the image).

• Each page will explain a little about the type of file, and any naming conventions that are required for the files.

• After opening a section and uploading files - let's say disturbances, a list of uploaded file names and their meta-data will be shown, as displayed in the wireframe. And a "Preview Generated Config" button can be shown for users to change any configs if needed. Same for classifiers, input db, and miscellaneous files.

• A problem with this is that currently, all the files are uploaded together to the backend on the /upload endpoint, and only then are the configs generated. This means that the users will have to go to all the four sections, upload files there.. and then come back to the previous sections to check the generated configs.

  • A possible solution here is to divide the /upload endpoint to 4 endpoints - /uploadDisturbances, /uploadClassifiers, /uploadDB, and /uploadMiscellaneous . After each upload, users can get the generated configs on the same page, and make any changes if needed.
  • Another solution is to have a single page for uploading all the files, and after uploading everything, we can have the 4 pages (like in the image), where they can edit the configs. They'll have to come back to the upload page if new files are to be uploaded though.

• I actually prefer the first option. On each page, they'd upload the respective file type and below get to see/edit the generated configs then and there. Since only classifiers are required, and disturbances, area boundaries are recommended but optional (according to this doc), user can visit the upload page for which they have the files available, and skip the others.

Also, currently, there is no endpoint for fetching and changing the config for a file. On the frontend, in the "Preview Generated Config" modal, the users can review the config, make changes and save. So, we would require an endpoint which can take the filename and provide the generated config, and also one to save the new file contents if they're changed.

After everything's uploaded and changes are made for the configs, the users can move to the Configure Parameters section (Local Domain, Modules, etc.) and make any changes if needed. At the Run Simulation, we would again need an endpoint for sending the changes in parameters config to the backend.

And since the simulation takes a long time (20 to 30mins), I was thinking we can e-mail the user the output's link. The user won't stay on the same page for that long.

All of this is open for discussion, if you think something can be changed please do put up your ideas.

CC: @aornugent

[Namyalg]

According to me, we can have:

/classifiers will upload classifiers
/misc for miscellaneous
/disturbance for disturbances

And finally an endpoint /uploadComplete to indicate that upload is done

So in the backend the same is happening
Instead of uploading all folders at once, on each endpoint that particular folder can be sent

[YashKandalkar]

the /classifiers, /misc, /disturbance endpoints can save the files and generate the configs on the appropriate route, what would the /uploadComplete endpoint do? after uploading the files on the previous endpoints, I think the user can directly go to Configure Parameters and then run the simulation. If they haven't uploaded any required files, they can be notified on the Run Simulation page.

[Namyalg]

The /uploadComplete is to perform the check that everything is uploaded as expected
Once this is confirmed, then they can configure parameters
It will help identifying errors earlier right?

[YashKandalkar]

Please go through this flow once, I think we wouldn't need that endpoint. Correct me I have misunderstood something. #330 (reply in thread)

[SanjaySinghRajpoot]

The first option will work the best, If a user has uploaded an invalid file then we can send a response for that particular file to be invalid. In case of uploading multiple files, the user has to again upload all the files. This can get complicated easily. Keeping different endpoints for different files will also enable us to make any changes in the backend if needed for a particular file. So according to we we can go ahead with the first option.

[YashKandalkar]

Exactly 💯

[padmajabhol]

Hi, thanks for opening the discussion, all of us can share our ideas here as we go on to develop the gcbm simulation editor.
I discussed with @aornugent about the any possible endpoints that we may use to verify/edit the input files in the backend.

/upload <- take any input and try to pre-process as much as possible
/files -> list files
/classifiers -> list classifiers
/disturbances -> list disturbances
/disturbances/id -> list file config
/disturbances/id/configure <- edit config

I think having separate endpoints to upload a particular input type might not be a good idea because adding all of them with the one that I mentioned above might be too much from the user's pov? I might be totally wrong here.

We can list files of a particular category if they are present in separate folders. Currently, we are cut pasting all input files in one folder(input/run4) right after they are pre-processed(in the upload endpoint). But we can do this step after the user has finished verifying/editing them.

cc @ankitaS11 @SanjaySinghRajpoot

[YashKandalkar]

Actually, the first option doesn't include any extra steps, as we would need 4 different dropboxes for uploading the different file types anyway. And after uploading users would need to visit the 4 editing pages for checking the generated configs (we can't have the editing for all the file types on the same page as it'd be too cluttered).
So, rather than having a different upload page, I was thinking on each page, they'd upload the respective file type and below that, they will get to see/edit the generated configs then and there.
They can choose not to visit the pages which they don't have the files for.
Everything will be in a single flow -

1. First the user will create a simulation,

   

2. Then, the user can upload the files. The required file types will be on the top with a red asterisk beside their name.
   Also, after uploading the files for a certain file type, they will be listed directly below the upload box, and will have a Preview Config button which, when clicked, will make a request to the server with the file name. (So, the server doesn't need to send the generated config of every file. It'll only send the requested config from the frontend). The Preview Config will open a modal which will have a GUI and a JSON editor side by side for editing the contents of the config files. After hitting Save, the new file contents will be sent to the server via an endpoint.

   

3. After the required files are uploaded, the Configure Parameters and the Run Simulation menu can be opened (as Configure Parameters already has sensible defaults, it does not require any changes. A simulation can be run without changing these (correct me if I'm wrong here).

   

@Namyalg Also, in this flow, I don't think there's a need for the /uploadComplete endpoint. What do you think?

[Namyalg]

@YashKandalkar In the current design, in the backend, there is a copy of files from the folders classifiers, disturbances etc to a single folder, and then these folders classifiers etc are deleted when the simulation is just about to start
For this particular step, i thought the /uploadComplete can be used
We can probably do this when the Run simulation is clicked and eliminate its need, i.e at the start of the dynamic endpoint

[YashKandalkar]

Oh! Yes, that sounds good!

[ankitaS11]

Hi everyone, just sharing what I had in mind:

Currently, disturbances, classifiers, db and miscellaneous - all are uploaded through one end-point call. Let’s also give a user separate end-points to upload these separately: /gcbm/upload/classifiers, /gcbm/upload/disturbances/, /gcbm/upload/miscellaneous. This specially helps when, for example, GCBM Belize will not have any disturbances, then users can just upload classifiers, miscellaneous separately.

The above will also help us solve the problem where we can support other runs like GCBM.Belize:

Runs without disturbances? Currently they error out, how can we fix this?
If the user has uploaded using /gcbm/upload/<> endpoints, and if they miss disturbances, that means they didn’t intend to run it for disturbances.

I like what has been suggested before in the discussion, and this is what I had in my mind:

It will probably be worth introducing more end-points, to allow users who need to know what are the configs before they want to edit, and again separately:

/list/all
/list/classifiers
/list/disturbances
/list/miscellaneous

# Or: /config/<>/view
/disturbances/config/view
/classifiers/config/view
/miscellaneous/config/view

# Or: /config/<>/edit
/disturbances/config/edit
/classifiers/config/edit
/miscellaneous/config/edit

[ankitaS11]

Hi everyone,

Apologies for the delay in response. This all sounds great, but I wanted to share some of my thoughts - please feel free to add/comment on any of these:

I guess, there are 2 types of users we can have:

1. Those who know development.
2. Those who don't.

My assumption is, that people from the 2nd category will use it more, for their needs and analysis - but we should also give an option to the users to have control over this API. So it will be worth it to support an endpoint that does everything for them (maybe 3 endpoints: start, upload and run -> which should show a progress bar and not just print the output message), and also multiple endpoints that divide the work into steps, and allow users to call them one by one if they want to debug what's happening.

I like the divide and conquer approach by @padmajabhol - and I also agree with @aornugent's comment, just wanted to add that we should also get rid of hard coding files as much as possible. I'm getting used to GCBM config files, so may not be an expert here, but one of the tasks should combine:

1. Refactor the code to help remove hard coding as much as possible.
2. The /dynamic endpoint uses a thread to launch the process, my suggestion would be to wait for the thread to finish and then return. A blocking process will make sure that nothing unexpected happens. We can also think of having a progress bar (tqdm maybe, or maybe rich) to show the process (15-20 minutes is a longer wait time).

Looking forward to your views on this. Thank you!

[YashKandalkar]

Hi, @ankitaS11 - I had a few doubts about this.

start, upload and run

I think we already have this setup right? A new simulation is created with a title, files are uploaded (we're thinking of splitting this into different parts but on the frontend it won't add any unnecessary steps for the user).
So I think we already have the setup ready for the 2nd type of user.

allow users to call them one by one if they want to debug what's happening.

Can you elaborate on this, please? I'm not sure if I get this.

Refactor the code to help remove hard coding as much as possible

Completely agreed 💯

wait for the thread to finish and then return. A blocking process will make sure that nothing unexpected happens

Won't the blocking code block the main server thread? If that happens then the server won't be able to listen to any other requests, I don't think that's desirable. I think if there are any bugs in the simulation run, the thread will return an error code.
I agree that we can have a progress bar though. I was also thinking that we can send an email to the user once the simulation is finished and a success code is returned. This way, the user won't need to stay on the same page for 20-30 minutes.
What do you think?

[Namyalg]

I agree with @ankitaS11 's idea of having both modes of operation
Can you elaborate more about the blocking process?
My idea was to run a python script as described here moja-global/FLINT.Cloud#117
to fetch the logs of the running container, and in case something went wrong (if the logs had words like error, failure etc) it can be reported. Will it be possible to run this script every 1-2 mins till the simulation successfully executes or stops due to failure ?

[ankitaS11]

Hi, @YashKandalkar

Great questions! Thanks for taking a look.

So I think we already have the setup ready for the 2nd type of user.

I’m sorry, I didn’t have much context/background about what’s existing in the frontend, but that’s great if it’s already there. What I meant was, currently we allow the users to upload all the files through one end-point, but my suggestion was to also allow them to upload files for each category separately. This will probably allow us to support the case where simulations can run without any disturbances. Currently, the end-point will raise an error if no disturbance is uploaded (in the /gcbm/upload endpoint), but let’s say if we allow this:

# This is valid
curl -F disturbances=”@disturbances/disturbances_2011_moja.tiff” -F title=”run4” http://localhost:8080/gcbm/disturbances/upload

# But if you run without any classifiers uploaded, it will fail and give you a nice error message.

This is also what I meant by “allow users to call them one by one if they want to debug what's happening.”

To address your concern regarding the thread, that’s a valid point - and I agree. @Namyalg - thank you! I agree that your solution is valid, and sounds better. I’ll have to take a better look at the PR for technical aspects, but looks like it has already got the approvals, so looking forward to it. :)

[Namyalg]

Thanks for the clarification @ankitaS11
I can work on any of the endpoints