Skip to content

Commit

Permalink
containerization of backend services (genshinsim#2265)
Browse files Browse the repository at this point in the history
* embed generator

* add embed image

* fix deployment yaml

* fix missing shell

* extra $ sign

* fix $ sign in wrong place

* try migrate to build-images

* add workflow to test on containers branch

* get rid of test prints

* fix build script metadata

* fix build script directory

* fix typos

* try a different image

* fix dockerfile + gitignore

* hack fix?

* add online check

* remove port check

* fix port check due to bug

* add assets service

* update protos and pipeline

* fix linting

* update embedgenerator checks and readme

* change containers deploy to release only
  • Loading branch information
srliao authored Nov 15, 2024
1 parent 1be975b commit 633e0e5
Show file tree
Hide file tree
Showing 92 changed files with 4,251 additions and 2,210 deletions.
52 changes: 52 additions & 0 deletions .github/actions/containers/embedgenerator/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: "Build embedgenerator container"
description: "build embed generator container"
inputs:
githubToken:
required: true
description: github token
githubActor:
required: true
description: github actor
githubRepo:
required: true
description: github repo

runs:
using: composite
steps:
- name: Build UI
working-directory: ./ui
shell: bash
run: yarn workspace @gcsim/embed build

- name: List UI dist
working-directory: ./ui/packages/embed/dist
shell: bash
run: ls -lh

- name: Log in to the Container registry
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
with:
registry: ghcr.io
username: ${{ inputs.githubActor }}
password: ${{ inputs.githubToken }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: ghcr.io/${{ inputs.githubRepo }}

- name: Build go executable
working-directory: ./cmd/services/embedgenerator
shell: bash
run: GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build .

- name: Build and push Docker image
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
with:
context: .
file: ./build/docker/embedgenerator/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
26 changes: 26 additions & 0 deletions .github/scripts/json-to-yaml.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@

import os
import json
import yaml

def json_to_yaml(subdir, file):
obj = None

json_file = os.path.join(subdir, file)
with open(json_file) as f:
obj = json.load(f)

yaml_file = os.path.join(subdir, "metadata.yaml")
with open(yaml_file, "w") as f:
yaml.dump(obj, f)

os.remove(json_file)


if __name__ == "__main__":

for subdir, dirs, files in os.walk("./containers"):
for f in files:
if f != "metadata.json":
continue
json_to_yaml(subdir, f)
202 changes: 202 additions & 0 deletions .github/scripts/prepare-matrices.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
#!/usr/bin/env python3
import importlib.util
import sys
import os

import json
import yaml
import requests

from subprocess import check_output

from os.path import isfile

# read repository owner's username from custom env vars, else read from GitHub Actions default env vars
repo_owner = os.environ.get('REPO_OWNER', os.environ.get('GITHUB_REPOSITORY_OWNER'))

TESTABLE_PLATFORMS = ["linux/amd64"]

CONTAINER_DIR = "./containers"

def load_metadata_file_yaml(file_path):
with open(file_path, "r") as f:
return yaml.safe_load(f)

def load_metadata_file_json(file_path):
with open(file_path, "r") as f:
return json.load(f)

def get_latest_version_py(latest_py_path, channel_name):
spec = importlib.util.spec_from_file_location("latest", latest_py_path)
latest = importlib.util.module_from_spec(spec)
sys.modules["latest"] = latest
spec.loader.exec_module(latest)
return latest.get_latest(channel_name)

def get_latest_version_sh(latest_sh_path, channel_name):
out = check_output([latest_sh_path, channel_name])
return out.decode("utf-8").strip()

def get_latest_version(subdir, channel_name):
ci_dir = os.path.join(subdir, "ci")
if os.path.isfile(os.path.join(ci_dir, "latest.py")):
return get_latest_version_py(os.path.join(ci_dir, "latest.py"), channel_name)
elif os.path.isfile(os.path.join(ci_dir, "latest.sh")):
return get_latest_version_sh(os.path.join(ci_dir, "latest.sh"), channel_name)
elif os.path.isfile(os.path.join(subdir, channel_name, "latest.py")):
return get_latest_version_py(os.path.join(subdir, channel_name, "latest.py"), channel_name)
elif os.path.isfile(os.path.join(subdir, channel_name, "latest.sh")):
return get_latest_version_sh(os.path.join(subdir, channel_name, "latest.sh"), channel_name)
return None

def get_published_version(image_name):
r = requests.get(
f"https://api.github.com/users/{repo_owner}/packages/container/{image_name}/versions",
headers={
"Accept": "application/vnd.github.v3+json",
"Authorization": "token " + os.environ["TOKEN"]
},
)

if r.status_code != 200:
return None

data = json.loads(r.text)
for image in data:
tags = image["metadata"]["container"]["tags"]
if "rolling" in tags:
tags.remove("rolling")
# Assume the longest string is the complete version number
return max(tags, key=len)

def run_build_sh(build_sh_path):
out = check_output([build_sh_path])
return out.decode("utf-8").strip()

def get_image_metadata(subdir, meta, hash, forRelease=False, force=False, channels=None):
imagesToBuild = {
"images": [],
"imagePlatforms": []
}

if channels is None:
channels = meta["channels"]
else:
channels = [channel for channel in meta["channels"] if channel["name"] in channels]


for channel in channels:
# Image Name
toBuild = {}
if channel.get("stable", False):
toBuild["name"] = meta["app"]
else:
toBuild["name"] = "-".join([meta["app"], channel["name"]])

# Skip if latest version already published
if not force:
published = get_published_version(toBuild["name"])
if published is not None and published == hash:
continue
toBuild["published_version"] = published

toBuild["version"] = hash

# Image Tags
toBuild["tags"] = ["rolling", hash]

# Platform Metadata
for platform in channel["platforms"]:

if platform not in TESTABLE_PLATFORMS and not forRelease:
continue

toBuild.setdefault("platforms", []).append(platform)

target_os = platform.split("/")[0]
target_arch = platform.split("/")[1]

platformToBuild = {}
platformToBuild["name"] = toBuild["name"]
platformToBuild["platform"] = platform
platformToBuild["target_os"] = target_os
platformToBuild["target_arch"] = target_arch
platformToBuild["version"] = hash
platformToBuild["channel"] = channel["name"]
# if platform == "linux/amd64":
# platformToBuild["builder"] = "ubuntu-latest"
# elif platform == "linux/arm64":
# #platformToBuild["builder"] = "arc-runner-set-containers-arm64"
platformToBuild["label_type"]="org.opencontainers.image"

# build scripts
if "build_scripts" in channel:
if platform in channel["build_scripts"]:
platformToBuild["build_script"] = os.path.join(subdir, channel["build_scripts"][platform])

if isfile(os.path.join(subdir, channel["name"], "Dockerfile")):
platformToBuild["dockerfile"] = os.path.join(subdir, channel["name"], "Dockerfile")
# platformToBuild["context"] = os.path.join(subdir, channel["name"])
platformToBuild["context"] = "./" # always use current dir as context
platformToBuild["goss_config"] = os.path.join(subdir, channel["name"], "goss.yaml")
else:
platformToBuild["dockerfile"] = os.path.join(subdir, "Dockerfile")
# platformToBuild["context"] = subdir
platformToBuild["context"] = "./" # always use current dir as context
platformToBuild["goss_config"] = os.path.join(subdir, "ci", "goss.yaml")

platformToBuild["goss_args"] = "tail -f /dev/null" if channel["tests"].get("type", "web") == "cli" else ""

platformToBuild["tests_enabled"] = channel["tests"]["enabled"] and platform in TESTABLE_PLATFORMS

imagesToBuild["imagePlatforms"].append(platformToBuild)
imagesToBuild["images"].append(toBuild)
return imagesToBuild

if __name__ == "__main__":
apps = sys.argv[1]
forRelease = sys.argv[2] == "true"
force = sys.argv[3] == "true"
hash = sys.argv[4]
imagesToBuild = {
"images": [],
"imagePlatforms": []
}

if apps != "all":
channels=None
apps = apps.split(",")
if len(sys.argv) == 5:
channels = sys.argv[4].split(",")

for app in apps:
if not os.path.exists(os.path.join(CONTAINER_DIR, app)):
print(f"App \"{app}\" not found")
exit(1)

meta = None
if os.path.isfile(os.path.join(CONTAINER_DIR, app, "metadata.yaml")):
meta = load_metadata_file_yaml(os.path.join(CONTAINER_DIR, app, "metadata.yaml"))
elif os.path.isfile(os.path.join(CONTAINER_DIR, app, "metadata.json")):
meta = load_metadata_file_json(os.path.join(CONTAINER_DIR, app, "metadata.json"))

imageToBuild = get_image_metadata(os.path.join(CONTAINER_DIR, app), meta, hash, forRelease, force=force, channels=channels)
if imageToBuild is not None:
imagesToBuild["images"].extend(imageToBuild["images"])
imagesToBuild["imagePlatforms"].extend(imageToBuild["imagePlatforms"])
else:
for subdir, dirs, files in os.walk(CONTAINER_DIR):
for file in files:
meta = None
if file == "metadata.yaml":
meta = load_metadata_file_yaml(os.path.join(subdir, file))
elif file == "metadata.json":
meta = load_metadata_file_json(os.path.join(subdir, file))
else:
continue
if meta is not None:
imageToBuild = get_image_metadata(subdir, meta, hash, forRelease, force=force)
if imageToBuild is not None:
imagesToBuild["images"].extend(imageToBuild["images"])
imagesToBuild["imagePlatforms"].extend(imageToBuild["imagePlatforms"])
print(json.dumps(imagesToBuild))
55 changes: 55 additions & 0 deletions .github/scripts/render-readme.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import os
import json
import requests
import yaml

from jinja2 import Environment, PackageLoader, select_autoescape

repo_owner = os.environ.get('REPO_OWNER', os.environ.get('GITHUB_REPOSITORY_OWNER'))
repo_name = os.environ.get('REPO_NAME', os.environ.get('GITHUB_REPOSITORY'))

env = Environment(
loader=PackageLoader("render-readme"),
autoescape=select_autoescape()
)

def load_metadata_file_yaml(file_path):
with open(file_path, "r") as f:
return yaml.safe_load(f)

def load_metadata_file_json(file_path):
with open(file_path, "r") as f:
return json.load(f)

def load_metadata_file(file_path):
if file_path.endswith(".json"):
return load_metadata_file_json(file_path)
elif file_path.endswith(".yaml"):
return load_metadata_file_yaml(file_path)
return None

if __name__ == "__main__":
app_images = []
for subdir, dirs, files in os.walk("./containers"):
for file in files:
if file != "metadata.yaml" and file != "metadata.json":
continue
meta = load_metadata_file(os.path.join(subdir, file))
for channel in meta["channels"]:
name = ""
if channel.get("stable", False):
name = meta["app"]
else:
name = "-".join([meta["app"], channel["name"]])
image = {
"name": name,
"channel": channel["name"],
"html_url": f"https://github.com/{repo_name}/pkgs/container/{name}",
"owner": repo_owner
}

app_images.append(image)

template = env.get_template("README.md.j2")
with open("./README.md", "w") as f:
f.write(template.render(app_images=app_images))
4 changes: 4 additions & 0 deletions .github/scripts/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
requests
pyyaml
packaging
jinja2
39 changes: 39 additions & 0 deletions .github/scripts/templates/README.md.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<!---
NOTE: AUTO-GENERATED FILE
to edit this file, instead edit its template at: ./github/scripts/templates/README.md.j2
-->
<div align="center">
## Containers
</div>

Welcome to gcsim container images. This registry contains containers for various gcsim services.

## Tag immutability

The containers built here do not use immutable tags, as least not in the more common way you have seen from [linuxserver.io](https://fleet.linuxserver.io/) or [Bitnami](https://bitnami.com/stacks/containers).

We do take a similar approach but instead of appending a `-ls69` or `-r420` prefix to the tag we instead insist on pinning to the sha256 digest of the image, while this is not as pretty it is just as functional in making the images immutable.

| Container | Immutable |
|----------------------------------------------------|-----------|
| `ghcr.io/genshinsim/embedgenerator:rolling` | ❌ |
| `ghcr.io/genshinsim/embedgenerator:3.0.8.1507` | ❌ |
| `ghcr.io/genshinsim/embedgenerator:rolling@sha256:8053...` | ✅ |
| `ghcr.io/genshinsim/embedgenerator:3.0.8.1507@sha256:8053...` | ✅ |

_If pinning an image to the sha256 digest, tools like [Renovate](https://github.com/renovatebot/renovate) support updating the container on a digest or application version change._


## Available Images

Each Image will be built with a `rolling` tag, along with tags specific to it's version. Available Images Below

Container | Channel | Image
--- | --- | ---
{% for image in app_images | sort(attribute="name") -%}
[{{ image.name }}]({{ image.html_url }}) | {{ image.channel }} | ghcr.io/{{ image.owner }}/{{ image.name }}
{% endfor %}

## Credits

This build script is based on the hard work of [onedr0p](https://github.com/onedr0p) and [joryirving](https://github.com/joryirving).
Loading

0 comments on commit 633e0e5

Please sign in to comment.