Dockerize analytics folder #1538

coilysiren · 2024-03-25T21:38:20Z

Summary

Time to review: 10 mins

Changes proposed

Changes the application to primarily be run inside of docker.

You can still run code natively via the PY_RUN_APPROACH=local setting, as described in the makefile. But docker is now the "default" method of running this code.

⚠️ Breaking Changes ⚠️

If you previously authed to Github via the web browser login flow, then you will now have to change to authing to Github via a personal access token. This is because I couldn't figure out how to get the web browser login flow working, but I could get the personal access token working.

To utilize the personal access token method:

Go to https://github.com/settings/tokens
Create a token
Give it the following scopes:
- repo
- read:org
- admin:public_key
- project
Set in your terminal via export GH_TOKEN=...

Testing

$ make sprint-reports-with-latest-data ACTION=post-results

=> Exporting project data from the sprint board
=====================================================
docker-compose run -e GH_TOKEN --rm grants-analytics poetry run analytics export gh_project_data \
	--owner HHS \
	--project 13 \
	--output-file data/sprint-data.json

coilysiren · 2024-03-25T22:32:57Z

analytics/Dockerfile

@@ -0,0 +1,113 @@
+# Use the official python3 image based on Debian 11 "Bullseye".


Mostly copied from api/Dockerfile

coilysiren · 2024-03-26T20:20:36Z

analytics/pyproject.toml

@@ -1,8 +1,8 @@
 [tool.poetry]
 authors = ["widal001 <billy.daly@agile6.com>"]
 description = "Python package for analyzing data related to the Simpler Grants Project"
-name = "analytics"
-packages = [{include = "analytics", from = "src"}]
+name = "simpler-grants-gov-analytics"


The poetry run pip install line in the Dockerfile references this name

coilysiren · 2024-03-26T20:21:02Z

analytics/docker-compose.yml

+    container_name: grants-analytics
+    volumes:
+      - .:/analytics
+      - ~/.ssh:/home/${RUN_USER:-analytics}/.ssh


The ssh key is for gh

coilysiren · 2024-03-26T20:22:09Z

analytics/Makefile

+# By default, all python/poetry commands will run inside of the docker container
+# if you wish to run this natively, add PY_RUN_APPROACH=local to your environment vars
+# You can set this by either running `export PY_RUN_APPROACH=local` in your shell or add
+# it to your ~/.zshrc file (and run `source ~/.zshrc`)
+ifeq "$(PY_RUN_APPROACH)" "local"
+POETRY := poetry run
+GITHUB := gh
+else
+POETRY := docker-compose run $(DOCKER_EXEC_ARGS) --rm $(APP_NAME) poetry run
+GITHUB := docker-compose run $(DOCKER_EXEC_ARGS) --rm $(APP_NAME) gh
+endif


Important to read the documentation here

Thanks for calling this out! Works like a charm:

@coilysiren can we update the development.md docs to reference this as well?

I added the same text verbatim to development.md 77139c1

coilysiren · 2024-03-26T20:31:36Z

analytics/Dockerfile

+# Set PYTHONPATH so that the tests can find the source code.
+ENV PYTHONPATH="/analytics/src/:$PYTHONPATH"


I had to add this line

Non-blocking request, but can we look into this further? If we're installing the analytics/ package in the docker container correctly, I don't think we should need to set the PYTHONPATH directly, see this stack overflow post for more info

Thanks for the feedback, but I'm not sure how valuable that would be right now, given that large set of work on my plate. I will make a ticket.

Here's the ticket: #1565

Yeah, I'm not super worried about it, mainly just want to flag it in case it resurfaces in some other error.

coilysiren · 2024-03-26T20:39:22Z

analytics/Dockerfile

+# Install gh CLI
+# docs: https://github.com/cli/cli/blob/trunk/docs/install_linux.md
+RUN mkdir -p -m 755 /etc/apt/keyrings && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
+  && chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
+  && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
+  && apt update \
+  && apt install gh -y \
+  && gh --version


This is new versus the api dockerfile

Awesome thanks for catching and handling the need to install the GitHub CLI

chouinar

Not at all familiar with this code, just a few minor notes

chouinar · 2024-03-27T13:34:49Z

analytics/Dockerfile

+  && apt-get install --no-install-recommends --yes \
+  build-essential \
+  libpq-dev \
+  postgresql \


Some of these apt-get values might be removable like this one if the analytics code never hits postgres

It is going to use postgres

build-essential: a good all around package to keep

libpq-dev: is for postgres

PostgreSQL: same as above

So, nothing to remove

chouinar · 2024-03-27T13:50:56Z

analytics/Dockerfile

+# Set the host to 0.0.0.0 to make the server available external
+# to the Docker container that it's running in.
+ENV HOST=0.0.0.0
+
+# Run the application.
+CMD ["poetry", "run", "python", "-m", "src"]


What is this running / what is the entrypoint file to this code?

The entrypoints are all of these commands here:

simpler-grants-gov/analytics/Makefile

Lines 51 to 90 in 5d898f9

sprint-data-export:

@echo "=> Exporting project data from the sprint board"

@echo "====================================================="

$(POETRY) run analytics export gh_project_data \

--owner $(ORG) \

--project $(SPRINT_PROJECT) \

--output-file $(SPRINT_FILE)

issue-data-export:

@echo "=> Exporting issue data from the repository"

@echo "====================================================="

$(POETRY) run analytics export gh_issue_data \

--owner $(ORG) \

--repo $(REPO) \

--output-file $(ISSUE_FILE)

gh-data-export: sprint-data-export issue-data-export

sprint-burndown:

@echo "=> Running sprint burndown report"

@echo "====================================================="

poetry run analytics calculate sprint_burndown \

--sprint-file $(SPRINT_FILE) \

--issue-file $(ISSUE_FILE) \

--sprint "$(SPRINT)" \

--unit $(UNIT) \

--$(ACTION)

percent-complete:

@echo "=> Running percent complete deliverable"

@echo "====================================================="

poetry run analytics calculate deliverable_percent_complete \

--sprint-file $(SPRINT_FILE) \

--issue-file $(ISSUE_FILE) \

--unit $(UNIT) \

--$(ACTION)

sprint-reports: sprint-burndown percent-complete

sprint-reports-with-latest-data: gh-data-export sprint-reports

I don't plan on specifying the entrypoint inside of this Dockerfile

I removed the lines you commented on

585c40e

widal001

LGTM!!

Tests

I ran both with Docker and natively and works well.

Natively:

With docker:

Non-blocking flag

At some point in the future, if our docker image is just using the latest python version we'll need to fix that deprecation warning for accessing the timezone via the 3.11 syntax.

widal001 · 2024-03-27T19:00:07Z

analytics/Dockerfile

+# Set PYTHONPATH so that the tests can find the source code.
+ENV PYTHONPATH="/analytics/src/:$PYTHONPATH"


Yeah, I'm not super worried about it, mainly just want to flag it in case it resurfaces in some other error.

widal001 · 2024-03-27T19:01:33Z

analytics/Dockerfile

+# The build stage that will be used to deploy to the various environments
+# needs to be called `release` in order to integrate with the repo's
+# top-level Makefile
+FROM python:3-slim AS base


Does this use 3.12 by default? If so we might want to update the minimum required python version in pyproject.toml and fix the warning around the use of 3.11-based timezone syntax, or change this to use 3.11.

Yeah its 3.12 right now

checkpoint-kai-1711402661

33b490b

github-actions bot added python analytics ci/cd labels Mar 25, 2024

coilysiren added 2 commits March 25, 2024 14:47

git restore

b53a249

add docker files

a07bafd

github-actions bot removed the python label Mar 25, 2024

checkpoint-kai-1711405786

5501a50

coilysiren commented Mar 25, 2024

View reviewed changes

coilysiren added 7 commits March 25, 2024 15:43

PYTHONPATH

86579db

ANALYTICS_SLACK_BOT_TOKEN

3fafefe

ANALYTICS_REPORTING_CHANNEL_ID

c2e4ff3

local.env

be474a1

install gh CLI

1e68afd

gh CLI login process working

132a3d2

workon env vars

1ea2610

coilysiren mentioned this pull request Mar 26, 2024

[Issue 1541] Ran make lint on analytics folder #1539

Merged

coilysiren added 2 commits March 26, 2024 11:49

Merge branch 'main' into analytics-docker

dabd384

Merge branch 'main' into analytics-docker

f062dfc

coilysiren commented Mar 26, 2024

View reviewed changes

Merge branch 'main' into analytics-docker

854a780

coilysiren commented Mar 26, 2024

View reviewed changes

coilysiren and others added 2 commits March 26, 2024 13:39

Update Dockerfile

acaa594

docs

2746bee

github-actions bot added the documentation Improvements or additions to documentation label Mar 26, 2024

coilysiren marked this pull request as ready for review March 26, 2024 22:22

coilysiren requested a review from sumiat as a code owner March 26, 2024 22:22

coilysiren requested review from widal001, andycochran, acouch, SammySteiner, aplybeah and chouinar as code owners March 26, 2024 22:22

chouinar reviewed Mar 27, 2024

View reviewed changes

remove copied code

585c40e

coilysiren requested a review from chouinar March 27, 2024 15:57

coilysiren and others added 2 commits March 27, 2024 11:41

add to development.md

77139c1

Merge branch 'main' into analytics-docker

51d8997

widal001 approved these changes Mar 27, 2024

View reviewed changes

coilysiren merged commit 0a25d0b into main Mar 27, 2024
2 checks passed

coilysiren deleted the analytics-docker branch March 27, 2024 19:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dockerize analytics folder #1538

Dockerize analytics folder #1538

coilysiren commented Mar 25, 2024 •

edited

Loading

coilysiren Mar 25, 2024 •

edited

Loading

coilysiren Mar 26, 2024

coilysiren Mar 26, 2024

coilysiren Mar 26, 2024

widal001 Mar 27, 2024

coilysiren Mar 27, 2024

coilysiren Mar 26, 2024

widal001 Mar 27, 2024

coilysiren Mar 27, 2024

coilysiren Mar 27, 2024

widal001 Mar 27, 2024

coilysiren Mar 26, 2024

widal001 Mar 27, 2024 •

edited

Loading

chouinar left a comment

chouinar Mar 27, 2024

coilysiren Mar 27, 2024

coilysiren Mar 27, 2024

chouinar Mar 27, 2024

coilysiren Mar 27, 2024

coilysiren Mar 27, 2024

widal001 left a comment •

edited

Loading

widal001 Mar 27, 2024

widal001 Mar 27, 2024 •

edited

Loading

coilysiren Mar 27, 2024

		@@ -0,0 +1,113 @@
		# Use the official python3 image based on Debian 11 "Bullseye".

		# Set PYTHONPATH so that the tests can find the source code.
		ENV PYTHONPATH="/analytics/src/:$PYTHONPATH"

	sprint-data-export:
	@echo "=> Exporting project data from the sprint board"
	@echo "====================================================="
	$(POETRY) run analytics export gh_project_data \
	--owner $(ORG) \
	--project $(SPRINT_PROJECT) \
	--output-file $(SPRINT_FILE)

	issue-data-export:
	@echo "=> Exporting issue data from the repository"
	@echo "====================================================="
	$(POETRY) run analytics export gh_issue_data \
	--owner $(ORG) \
	--repo $(REPO) \
	--output-file $(ISSUE_FILE)

	gh-data-export: sprint-data-export issue-data-export

	sprint-burndown:
	@echo "=> Running sprint burndown report"
	@echo "====================================================="
	poetry run analytics calculate sprint_burndown \
	--sprint-file $(SPRINT_FILE) \
	--issue-file $(ISSUE_FILE) \
	--sprint "$(SPRINT)" \
	--unit $(UNIT) \
	--$(ACTION)

	percent-complete:
	@echo "=> Running percent complete deliverable"
	@echo "====================================================="
	poetry run analytics calculate deliverable_percent_complete \
	--sprint-file $(SPRINT_FILE) \
	--issue-file $(ISSUE_FILE) \
	--unit $(UNIT) \
	--$(ACTION)

	sprint-reports: sprint-burndown percent-complete

	sprint-reports-with-latest-data: gh-data-export sprint-reports

Dockerize analytics folder #1538

Dockerize analytics folder #1538

Conversation

coilysiren commented Mar 25, 2024 • edited Loading

Summary

Time to review: 10 mins

Changes proposed

⚠️ Breaking Changes ⚠️

Testing

coilysiren Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

widal001 Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

chouinar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

widal001 left a comment • edited Loading

Choose a reason for hiding this comment

Tests

Non-blocking flag

Choose a reason for hiding this comment

widal001 Mar 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coilysiren commented Mar 25, 2024 •

edited

Loading

coilysiren Mar 25, 2024 •

edited

Loading

widal001 Mar 27, 2024 •

edited

Loading

widal001 left a comment •

edited

Loading

widal001 Mar 27, 2024 •

edited

Loading