Automate Config Creation

[Abby-Wheelis]

As the number of programs using our platform continues to increase, one of the time consuming points in the startup process is the development, verification, and editing of the config files. I plan to add a feature to nrel-openpath-deploy-configs in order to facilitate the creation of these config files. In discussing this idea, @shankari mentioned that we could leverage GitHub pages in order to allow partners to fill in a form, and submit it to create a PR into that repo. With this route there are a few things that I think we should keep in mind:

• how to verify the configs - we need automation to help ensure that the configs are valid before they "go live"
• how to edit the configs later - partners often change their mind about various options in the config, so editing also needs to be part of this process
• how will the new PRs be merged? - will we still have to merge each config (and edit) PR by hand? - what other options might we have

I also believe that the more user-friendly and well-validated this interface is, the more people will be able to set the configs up themselves, improving efficiency of the startup workflow and saving us time.

[shankari]

@Abby-Wheelis here's the example that I saw (while wearing my JOET hat, incidentally)
https://landscape.lfenergy.org/

You can see that it has links at the top to add and update entries.
https://github.com/lf-energy/lfenergy-landscape/issues/new?assignees=jmertic%2CYarille&labels=new+entry&template=new-landscape-entry.yml&title=New+Landscape+Entry+-+%5BPROJECT%5D

Not sure what happens when the issue is filed - you should be able to look at the commit history and figure out what they do. Maybe it is more manual than I originally thought.

[shankari]

There is a way to run a workflow when an issue is created
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#issues
so you could have people create an issue and then trigger a workflow that validates it and creates a PR
or something cool and automated like that 😄

[Abby-Wheelis]

The form for adding a new entry is an "issue template" which seems to be a beta feature on github, I agree that it would be great if we could do something like that - snooping through their workflow, it looks like the PR is mostly created by automation - there's two commits from an Auto-Updater bot and then a comment from another bot with a deployment preview. Based on this, I think the first flow that I intend to try would be to set up an issue form, have the creation of that issue generate a pull request, and set up a check on the pull request to make sure the config is valid. I believe that would leave our only step as actually merging the pull request, which I'm sure we could automate later as well, it seems the repo you linked already does this.

[Abby-Wheelis]

I've started creating an issue template, and I think it looks good so far! Before I keep going through the rest of the form, I plan to test out the form and see what I am able to do in terms of processing the form output into a config file.

[shankari]

I believe that would leave our only step as actually merging the pull request, which I'm sure we could automate later as well, it seems the repo you linked already does this.

I think we want to keep at least one manual approval step to avoid attacks in which bad actors create configs (threat: denial of service through spamming), or worse, edit existing configs to introduce errors (threat: hijacking the server that data goes to, for example).

[Abby-Wheelis]

I was beginning to get stuck in terms of figuring out how to convert the issue into a file and got a chance to chat with @nataliejschultz today, where she helped me figure out some more steps of the workflow - including going from issue created to creating a PR with something like this: create pull request action. While we were talking, I also managed to find github-issue-to-json-file, a repo that can be used to create a json file from an issue, and proved that we can complete this process. It also uses the issue templates, so I'm encouraged that we're on the right track. I don't see too much activity on that repository, so I'm not sure if we'll want to use it directly but it does serve as a proof of concept - I hadn't really found any cases of issue -> file conversion before I found that repo, but it seemed like it had to be an existing use case somewhere.

[Abby-Wheelis]

I just tested the workflow from that Repo - and it does work! (Ignore the mismatch, it's because I keep changing things). That workflow automatically creates a file with a randomized name, and I know we'll want our file to have a specific name, I also note that the this creates a "flat" JSON file, and we need ours to have a specific structure, for those reasons, my current workflow idea would consist of a few actions:

• on issue creation, edit, etc: create the config file with the converter
• process the generated JSON file -> JSON file that fits our needs (I would write a custom script for this)
• create a PR to add the processed json file to our configs folder
• run checks on the PR to ensure the config is valid
• manually approve and merge config

This process does hinge on the issue -> json generator that I found. @shankari can you take a look when you get a chance and see how you feel about using it as a dependency? (repo here)

[shankari]

@Abby-Wheelis given that the generator doesn't do the exact thing that you need, and, as you point out, might not be actively maintained, why don't you write your own GitHub action? You can use https://github.com/zachleat/github-issue-to-json-file as a model. Writing GitHub action is not that different from writing a workflow - I think it uses javascript under the hood.

Rest of the workflow seems sound.

As a future enhancement, might I suggest:

• running some automated phone tests as part of config validation
• applying the config to the relevant sites (mainly join page + app) and storing the resulting "previews" in the PR.

But that is definitely way more complicated, so we shouldn't hold this up for now

[Abby-Wheelis]

I had not really parsed how the generator works at first, but going back there were just 3 main files I needed! I now have this up and running, converting an issue to json and saving it to the appropriate folder. My next step is going to be working through the formatting piece, as it stands the json file is still "flat". I think I've included most of the fields in the template now, but may edit as needed to facilitate parsing into the desired format.

[Abby-Wheelis]

Starting to structure the json file...

I plan to make sure the code parsing the issue is set up with good error messages, so if the workflow fails, the creator knows what to fix

[Abby-Wheelis]

Working on my github - creating a pull request!

[Abby-Wheelis]

Something @sebastianbarry brought up today which was a great point, is what happens when the config file changes, such as adding a new field or way to customize? This will likely involve updating the issue template, the workflow, and possibly any validation step I set up. With this in mind, I need to be sure that I keep up with documentation for this, something like a "how to update automatic config creation" document might be a good idea.

I've been focused on what happens if the deployer needs to update their config (I think updating their existing issue or filing a new one would both do the trick!), but I should keep in mind developer-side updates as well.

[Abby-Wheelis]

I have added a different dependency to this workflow, one that creates pull requests. Unlike the other outside action I was considering using, this one does exactly what we want it to, is widely used (95k users), and is actively maintained (commits within the last month), and author is active in responding to issues. The documentation is also extensive. See the repo here

[Abby-Wheelis]

I have left a few fields hard-coded, rather than allowing the person filling out the form to set them:

• aggregate_call_auth -> every deployment so far has user_only so I hardcoded that
  • what does this control?
  • searching all e-mission code shows some sample test/files with no_auth, would it make sense to offer a dropdown between user_only and no_auth, or keep it as is with user_only hardcoded?
• token_prefix -> hardcoded as nrelop
  • are there any cases where it would not be nrelop (for a deployment)? This is one of the checks we perform at login, so another prefix would not allow for login I'm fairly sure.
• support_upload -> at least for now, this has to be false in production, so I hardcoded it as such
  • enabling would require additional approvals
• Is there anything else that should not be able to be changed?

@shankari in your opinion, any of these I should not hardcode (or others I should)?

[shankari]

@Abby-Wheelis I think that is fine. The only exception may be the token_prefix in case any of the other deployers want to use the same system and have their own prefix. But we should not over-engineer at this point; let's wait until the use case actually shows up before addressing it. We can always iterate again later if we find any other manual steps.

[Abby-Wheelis]

I'm a little stumped in a couple of places:

1. configuring "highly custom" features - this specifically applies to custom trip surveys, timeuse surveys, and reminder schemes in my mind, as they have a relatively high number of options. What I've done for now is to include a field where deployers can describe what they are looking to do for each of these, for reminders, this should mean that someone can then add the code to the PR in just a couple of minutes. Timeuse might require some back and forth with obtaining surveys and setting up labels. For regular trip surveys - I added a field for the path, and intend to set the labels to the baseline "answered" -- if we want to go back and customize the labels, this will take some manual work
2. testing/validating the config once generated:
   • the generation process does some validating, as it handles parsing the issue and formatting the file
   • the form has required fields and defaults set in place to prevent missing fields
   • I don't know how to truly test the config after it's generated
     • I tend to make an instance of the opcode in my local server, start the dev app, and log in with the opcode I mad
     • this is time consuming and has a lot of steps, so is not sustainable as a solution long-term
     • the "future enhancements" suggested above would be ideal probably
     • even if I did spin up a local server, create an opcode, and spin up the devapp, I'm not sure how I would replicate the logging in and clicking through onboarding that I normally do to check on things
   • things I'm normally checking when I do it by hand:
     • that the app does not crash
     • that summary and protocol text looks fine
     • if custom onboarding survey, that it populates
     • if custom labels, that they show up
     • if trip survey, that it shows up and can be filled

The workflow I have so far seems to be working great for configs that are not "highly custom", and succeeding in eliminating the copy-paste process that we currently need for new configs. The automation means formatting and convention are consistent, but if something slips past the automation I still want to catch it before merging. I'll keep brainstorming and researching...

[shankari]

@Abby-Wheelis I think that the perfect should not be the enemy of the good. I would suggest finishing up a PR with this flow and with some (clearly documented) manual steps. We should then demo it to the contracting folks and make sure that they are on board, and/or integrate with @nataliejschultz's code.

We can then work on the enhancements step by step (I have ideas). Incremental progress with lots of feedback is the best way to approach these customer facing changes.

[Abby-Wheelis]

I have a PR started here, and iterating on this step first I think the only remaining task is to work on documentation: I've added a note to the existing deploy-configs readme and am planning a PR to add 2 docs to e-mission-docs, one as a guide to partners and one as a guide to developers, and linking them with that note I added to the deploy-configs readme.

[Abby-Wheelis]

To keep this issue updated, we have implemented the process and are continuing with polishing the process and getting it approved before we start using it.

Most recent PR: e-mission/nrel-openpath-deploy-configs#49

[Abby-Wheelis]

This process is now in place and has been used a few times! Some partners have been confused by the fact that they need to make an account on GitHub, so we may need to improve communication around how github works (apparently the space themed landing page can seem scammy), but the issue -> PR process seems to work well otherwise.