GitHub Backup!

An App that helps you backup your GitHub Issues so that you can still work when (you/they are) offline.

Why?

As a person (team of people) that uses GitHub as their "single-source-of-truth"
I need a backup of GitHub issues
so that I can see issue history and/or work "offline".
(either when I have no network or GH is "temporarily offline")

What?

GitHub (Issue) Backup is an App that lets you (unsurprisingly):

1. Store a backup of the content of the GitHub issue(s) for a given project/repo.
2. Track the changes/edits made to issue description and comments. see: dear-github/issues/129
3. View all your issues when you or GitHub are "offline"

How?

The app is built using the Phoenix Web Framework
(using the Elixir programming language)
and Tachyons UI ("Functional CSS") system.
While you can run the App without knowing Elixir or Phoenix
(see instructions below), if you want to understand how it works,
we recommend taking the time to learn each item on the list.
If you are new to Phoenix, Elixir, Tachyons (the UI) or ngrok,
we have a "beginner tutorials" which will bring you up-to-speed:

• Elixir: github.com/dwyl/learn-elixir
• Phoenix: github.com/dwyl/learn-phoenix-framework
• Tachyons: github.com/dwyl/learn-tachyons
• ngrok: github.com/dwyl/learn-ngrok

Additionally we use Amazon Web Services (AWS) Simple Storage Service (S3)
for storing the issue comment history with "high availability" and reliability.
To run github-backup on your localhost you will need to have an AWS account and an S3 "bucket".
(instructions given below).

Set Up Checklist on localhost

This will take approximately 15 minutes to complete.
(provided you already have a GitHub and AWS account)

Install Dependencies

To run the App on your localhost, you will need to have 4 dependencies installed:

• Elixir: https://github.com/dwyl/learn-elixir#installation
• PostgreSQL: https://github.com/dwyl/learn-postgresql#installation
• Phoenix: https://hexdocs.pm/phoenix/installation.html
• ngrok: https://github.com/dwyl/learn-ngrok#1-download--install
  (so you can share the app on your localhost with GitHub via public URL)

Once all the necessary dependencies are installed, we can move on. (if you get stuck, please let us know!)

Clone the App from GitHub

Run the following the command to clone the app from GitHub:

git clone https://github.com/dwyl/github-backup.git && cd github-backup

Required Environment Variables

To run the project on your localhost, you will need to have the following Environment Variables defined.

Note: if you are new to "Environment Variables",
please read: github.com/dwyl/learn-environment-variables

• APP_HOST - the hostname for your app e.g: localhost or gitbu.io.
• PRIVATE_KEY - The RSA private key for your GitHub App. See below for how to set this up.
• GITHUB_APP_ID - The unique id of your GitHub App. See below.
• GITHUB_APP_NAME - the name of your GitHub App. See below (Step 1).
• SECRET_KEY_BASE - a 64bit string used by Phoenix for security (to sign cookies and CSRF tokens). See below for how to generate yours.
• AWS_ACCESS_KEY_ID - your AWS access key (get this from your AWS settings)
• AWS_SECRET_ACCESS_KEY - your AWS secret access key
• S3_BUCKET_NAME - name of the AWS S3 "bucket" where issue comments will be stored.

Copy The .env_sample File

The easy way manage your Environment Variables locally is to have a .env file in the root of the project.

Copy the sample one and update the variables:

cp .env_sample .env

Now update the values to the real ones for your App.
You will need to register a GitHub app, so that is what we will do below!

Don't worry, your sensitive variables are safe as the .env file is "ignored" in the .gitignore file

Create a GitHub Application

The role of the Github application is to send notifications when events occur in your repositories.
For example you can be notified when new issues or pull requests are opened.

Note: we have simplified the steps in the "official" Github guide to create a new Github App: https://developer.github.com/apps/building-github-apps/creating-a-github-app

1. Access the New Application Settings Page

While logged into your GitHub Account, visit: https://github.com/settings/apps/new
Or navigate the following menus:

Settings -> Developer settings -> Github Apps -> New Github App

1. Github App name: The name of the app; must be unique, so can't be "gh-backup" as that's taken!
2. Descriptions: A short description of the app; "My backup app"
3. Homepage URL: The website of the app. e.g: "https://gitbu.io" (yours will be different)
4. User authorization callback URL: Redirect url after user authentication e.g."http://localhost:4000/auth/github/callback". This is not needed for backup so this field can be left empty.
5. Setup URL (optional): Redirect the user to this url after installation, not needed for github-backup.
6. Webhook URL: the URL where HTTP POST requests from Github are sent. The endpoint in the github-backup App is /event/new, however Github won't be able to send requests to http://localhost:4000/event/new as this url is only accessible on your localhost. To allow GitHub to access your localhost server you can use ngrok. Remember to update these value after you have a running server on your machine!
7. Webhook secret (optional) - leave this blank for now. see: github-backup/issues/76 (please let us know your thoughts on how/when we should prioritise this!)

2. Set the Necessary Permissions

Still on the same page, scroll down below the initial setup, to the Permissions section:

Define the access rights for the application on the permission section.

1. Change "issues" to "Read & write" (this is required to update "meta table" which links to the "issue history")

2. Change "Pull requests" to "Read-only" (this is required to allow the app to see the current state of pull requests)

3. Subscribe to Events

Scroll down to the "Subscribe to events" section and check the boxes for Issue comment and Issues:

4. Where can this GitHub App be installed?

Scroll down to the "Where can this GitHub App be installed?" section.

In our case we want to be able to use GitHub Backup for a number of projects in different organisations, so we selected Any account. but if you only want to backup your personal projects, then select Only on this account.

5. Click on the Create GitHub App Button!

Finally, with everything configured, click on "Create Github App"!

You should see a message confirming that your app was created:

(obviously your app will will have a different name in the url/path...)

6. Generate Private Key

Click on the link to "generate a private key".

When you click the Generate private key button it should download a file containing your private key.

Open the file in your text editor, e.g:

Don't worry, this is not our "real" private key ... it's just for demonstration purposes.
but this is what your RSA private key will look like, a block of random characters ...

The downloaded file contains the private key. Save this file in the root of the github-backup repo on your localhost.

For example in our case the GitHub app is called "gitbu",
so our Private Key file starts with the app name and the date the key was generated.
e.g: gitbu.2018-03-23.private-key.pem

Don't worry, all .pem (private key) files are ignored in the .gitignore file so your file will stay private.

Once you have copied the file into your project root, run the following command:

echo "export PRIVATE_KEY='`cat ./gitbu.2018-03-23.private-key.pem`'" >> .env

Replace the gitbu.2018-03-23.private-key.pem part in the command with the name of your private key file e.g: my-github-backup-app.2018-04-01.private-key.pem

That will create an entry in the .env file for your PRIVATE_KEY environment variable.

7. Copy the App Name and App ID from GitHub App Settings Page

In the "Basic information" section of your app's settings page, copy the GitHub App name and paste it into your .env file after the = sign for the GITHUB_APP_NAME variable.

Scroll down to the "About" section where you will find the ID of your app. This is the number that needs to be set for GITHUB_APP_ID in your .env file.

e.g: 103

Generate the SECRET_KEY_BASE

Run the following command to generate a new phoenix secret key:

mix phx.gen.secret

copy-paste the output (64bit String) into your .env file after the "equals sign" on the line for SECRET_KEY_BASE:

export SECRET_KEY_BASE=YourSecretKeyBaseGeneratedUsing-mix_phx.gen.secret

S3 Bucket

In order to save the Issue Comments (which are a variable length "rich text"), we save these as .json in an S3 bucket. If you do not already have an S3 bucket where you can store issues/comments, create one now.

Once you have an S3 bucket, copy the name and paste it into your .env file against the S3_BUCKET_NAME environment variable.

You will also need to define the

"Source" the .env File

Now that you have defined all the necessary environment variables in your .env file,
it's time to make them available to the App.

Execute the command following command in your terminal:

source .env

Note: This method only adds the environment variables locally and temporarily
so you need to start your server in the same terminal where you ran the source command.

Install Elixir (Application-specific) Dependencies

Now that you have the environment variables defined, you can install the elixir (application-specific) dependencies:

mix deps.get && cd assets && npm install && cd ..

Run the Tests! (To check it works!)

Confirm everything is working by running the tests:

mix test

Create the Database

In your terminal, run the following command:

mix ecto.create && mix ecto.migrate

Note: your PostgreSQL server will need to be running for this to work.

Run a github-backup (Phoenix Application) Server

The github-backup server will receive events from Github and then save the details or edits to an external database.

mix phx.server

Run ngrok on your localhost

In a New Window/Tab in your Terminal run the ngrok server:

ngrok http 4000

You should see something like this:

On the line that says "forwarding" you will see the (public) URL of your app, in this case https://38e239f0.ngrok.io (your URL's subdomain will be a different "random" set of characters)

Copy the URL for the next step. (remember to copy the https version)

Update your Webhook URL on your GitHub App

Now that your server is running you can update the webhook url in your app config.

You can read more about webhooks and ngrok at: https://developer.github.com/webhooks/configuring

On the "GitHub App settings page" Find the "Webhook URL" section: (the URL is currently http://localhost:4000 ...)

Replace the http://localhost:4000 with your unique ngrok url e.g:

Scroll down and click on the "Save changes" button:

NOTE: you will need to update the webhook URL each time you disconnect/connect to ngrok because a different URL is generated every time.

• copy and save the url into your Github App config with /event/new for endpoint

View the App in your Web Browser

Open http://localhost:4000 in your web browser. You should see something like:

Try The App!

To test your github-backup server try installing your app onto one of your repos.

1. Visit Your App's GitHub Page

Visit your App's URL on GitHub e.g: https://github.com/apps/gitbu

Click on the Install button to install the App for your organisation / repository.

2. Install the App

Select the Organisation you wish to install the app for:

Select the repository you want to install the App for e.g:

then click Install.

You should now see a settings page confirming that the app is installed:

3. Create/Update or Close an Issue in the Repo

In the repo you added the App to, create/update or close an issue. e.g:

Since this issue has/had already been resolved by a Pull Request from Sam, I closed it with a comment:

4. Observe the Data in Phoenix App Terminal Window

When you perform any Issue action (create, comment, update or close), you will see a corresponding event in the terminal.

You should see the payload of the request in the tab you have open in your terminal for the phoenix server, e.g:

5. Confirm the Data in PostgreSQL Database

You can confirm that the data was saved to PostgreSQL by checking your DB.
Open pgAdmin and navigate to the database table in app_dev > tables > issues

To run the query, right-click (⌘ + click) on the table (in this case issues) and then click "View/Edit Data" followed by "All Rows":

You can also confirm that the status of the issue has been update to "closed":

That's it, your github-backup instance is working!

Deployment

The project is hosted on Heroku at: https://github-backup.herokuapp.com

If you want to know how see:
https://github.com/dwyl/learn-heroku/blob/master/elixir-phoenix-app-deployment.md