forked from genshinsim/gcsim
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
containerization of backend services (genshinsim#2265)
* embed generator * add embed image * fix deployment yaml * fix missing shell * extra $ sign * fix $ sign in wrong place * try migrate to build-images * add workflow to test on containers branch * get rid of test prints * fix build script metadata * fix build script directory * fix typos * try a different image * fix dockerfile + gitignore * hack fix? * add online check * remove port check * fix port check due to bug * add assets service * update protos and pipeline * fix linting * update embedgenerator checks and readme * change containers deploy to release only
- Loading branch information
Showing
92 changed files
with
4,251 additions
and
2,210 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
name: "Build embedgenerator container" | ||
description: "build embed generator container" | ||
inputs: | ||
githubToken: | ||
required: true | ||
description: github token | ||
githubActor: | ||
required: true | ||
description: github actor | ||
githubRepo: | ||
required: true | ||
description: github repo | ||
|
||
runs: | ||
using: composite | ||
steps: | ||
- name: Build UI | ||
working-directory: ./ui | ||
shell: bash | ||
run: yarn workspace @gcsim/embed build | ||
|
||
- name: List UI dist | ||
working-directory: ./ui/packages/embed/dist | ||
shell: bash | ||
run: ls -lh | ||
|
||
- name: Log in to the Container registry | ||
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1 | ||
with: | ||
registry: ghcr.io | ||
username: ${{ inputs.githubActor }} | ||
password: ${{ inputs.githubToken }} | ||
|
||
- name: Extract metadata (tags, labels) for Docker | ||
id: meta | ||
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7 | ||
with: | ||
images: ghcr.io/${{ inputs.githubRepo }} | ||
|
||
- name: Build go executable | ||
working-directory: ./cmd/services/embedgenerator | ||
shell: bash | ||
run: GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build . | ||
|
||
- name: Build and push Docker image | ||
uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4 | ||
with: | ||
context: . | ||
file: ./build/docker/embedgenerator/Dockerfile | ||
push: true | ||
tags: ${{ steps.meta.outputs.tags }} | ||
labels: ${{ steps.meta.outputs.labels }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
|
||
import os | ||
import json | ||
import yaml | ||
|
||
def json_to_yaml(subdir, file): | ||
obj = None | ||
|
||
json_file = os.path.join(subdir, file) | ||
with open(json_file) as f: | ||
obj = json.load(f) | ||
|
||
yaml_file = os.path.join(subdir, "metadata.yaml") | ||
with open(yaml_file, "w") as f: | ||
yaml.dump(obj, f) | ||
|
||
os.remove(json_file) | ||
|
||
|
||
if __name__ == "__main__": | ||
|
||
for subdir, dirs, files in os.walk("./containers"): | ||
for f in files: | ||
if f != "metadata.json": | ||
continue | ||
json_to_yaml(subdir, f) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,202 @@ | ||
#!/usr/bin/env python3 | ||
import importlib.util | ||
import sys | ||
import os | ||
|
||
import json | ||
import yaml | ||
import requests | ||
|
||
from subprocess import check_output | ||
|
||
from os.path import isfile | ||
|
||
# read repository owner's username from custom env vars, else read from GitHub Actions default env vars | ||
repo_owner = os.environ.get('REPO_OWNER', os.environ.get('GITHUB_REPOSITORY_OWNER')) | ||
|
||
TESTABLE_PLATFORMS = ["linux/amd64"] | ||
|
||
CONTAINER_DIR = "./containers" | ||
|
||
def load_metadata_file_yaml(file_path): | ||
with open(file_path, "r") as f: | ||
return yaml.safe_load(f) | ||
|
||
def load_metadata_file_json(file_path): | ||
with open(file_path, "r") as f: | ||
return json.load(f) | ||
|
||
def get_latest_version_py(latest_py_path, channel_name): | ||
spec = importlib.util.spec_from_file_location("latest", latest_py_path) | ||
latest = importlib.util.module_from_spec(spec) | ||
sys.modules["latest"] = latest | ||
spec.loader.exec_module(latest) | ||
return latest.get_latest(channel_name) | ||
|
||
def get_latest_version_sh(latest_sh_path, channel_name): | ||
out = check_output([latest_sh_path, channel_name]) | ||
return out.decode("utf-8").strip() | ||
|
||
def get_latest_version(subdir, channel_name): | ||
ci_dir = os.path.join(subdir, "ci") | ||
if os.path.isfile(os.path.join(ci_dir, "latest.py")): | ||
return get_latest_version_py(os.path.join(ci_dir, "latest.py"), channel_name) | ||
elif os.path.isfile(os.path.join(ci_dir, "latest.sh")): | ||
return get_latest_version_sh(os.path.join(ci_dir, "latest.sh"), channel_name) | ||
elif os.path.isfile(os.path.join(subdir, channel_name, "latest.py")): | ||
return get_latest_version_py(os.path.join(subdir, channel_name, "latest.py"), channel_name) | ||
elif os.path.isfile(os.path.join(subdir, channel_name, "latest.sh")): | ||
return get_latest_version_sh(os.path.join(subdir, channel_name, "latest.sh"), channel_name) | ||
return None | ||
|
||
def get_published_version(image_name): | ||
r = requests.get( | ||
f"https://api.github.com/users/{repo_owner}/packages/container/{image_name}/versions", | ||
headers={ | ||
"Accept": "application/vnd.github.v3+json", | ||
"Authorization": "token " + os.environ["TOKEN"] | ||
}, | ||
) | ||
|
||
if r.status_code != 200: | ||
return None | ||
|
||
data = json.loads(r.text) | ||
for image in data: | ||
tags = image["metadata"]["container"]["tags"] | ||
if "rolling" in tags: | ||
tags.remove("rolling") | ||
# Assume the longest string is the complete version number | ||
return max(tags, key=len) | ||
|
||
def run_build_sh(build_sh_path): | ||
out = check_output([build_sh_path]) | ||
return out.decode("utf-8").strip() | ||
|
||
def get_image_metadata(subdir, meta, hash, forRelease=False, force=False, channels=None): | ||
imagesToBuild = { | ||
"images": [], | ||
"imagePlatforms": [] | ||
} | ||
|
||
if channels is None: | ||
channels = meta["channels"] | ||
else: | ||
channels = [channel for channel in meta["channels"] if channel["name"] in channels] | ||
|
||
|
||
for channel in channels: | ||
# Image Name | ||
toBuild = {} | ||
if channel.get("stable", False): | ||
toBuild["name"] = meta["app"] | ||
else: | ||
toBuild["name"] = "-".join([meta["app"], channel["name"]]) | ||
|
||
# Skip if latest version already published | ||
if not force: | ||
published = get_published_version(toBuild["name"]) | ||
if published is not None and published == hash: | ||
continue | ||
toBuild["published_version"] = published | ||
|
||
toBuild["version"] = hash | ||
|
||
# Image Tags | ||
toBuild["tags"] = ["rolling", hash] | ||
|
||
# Platform Metadata | ||
for platform in channel["platforms"]: | ||
|
||
if platform not in TESTABLE_PLATFORMS and not forRelease: | ||
continue | ||
|
||
toBuild.setdefault("platforms", []).append(platform) | ||
|
||
target_os = platform.split("/")[0] | ||
target_arch = platform.split("/")[1] | ||
|
||
platformToBuild = {} | ||
platformToBuild["name"] = toBuild["name"] | ||
platformToBuild["platform"] = platform | ||
platformToBuild["target_os"] = target_os | ||
platformToBuild["target_arch"] = target_arch | ||
platformToBuild["version"] = hash | ||
platformToBuild["channel"] = channel["name"] | ||
# if platform == "linux/amd64": | ||
# platformToBuild["builder"] = "ubuntu-latest" | ||
# elif platform == "linux/arm64": | ||
# #platformToBuild["builder"] = "arc-runner-set-containers-arm64" | ||
platformToBuild["label_type"]="org.opencontainers.image" | ||
|
||
# build scripts | ||
if "build_scripts" in channel: | ||
if platform in channel["build_scripts"]: | ||
platformToBuild["build_script"] = os.path.join(subdir, channel["build_scripts"][platform]) | ||
|
||
if isfile(os.path.join(subdir, channel["name"], "Dockerfile")): | ||
platformToBuild["dockerfile"] = os.path.join(subdir, channel["name"], "Dockerfile") | ||
# platformToBuild["context"] = os.path.join(subdir, channel["name"]) | ||
platformToBuild["context"] = "./" # always use current dir as context | ||
platformToBuild["goss_config"] = os.path.join(subdir, channel["name"], "goss.yaml") | ||
else: | ||
platformToBuild["dockerfile"] = os.path.join(subdir, "Dockerfile") | ||
# platformToBuild["context"] = subdir | ||
platformToBuild["context"] = "./" # always use current dir as context | ||
platformToBuild["goss_config"] = os.path.join(subdir, "ci", "goss.yaml") | ||
|
||
platformToBuild["goss_args"] = "tail -f /dev/null" if channel["tests"].get("type", "web") == "cli" else "" | ||
|
||
platformToBuild["tests_enabled"] = channel["tests"]["enabled"] and platform in TESTABLE_PLATFORMS | ||
|
||
imagesToBuild["imagePlatforms"].append(platformToBuild) | ||
imagesToBuild["images"].append(toBuild) | ||
return imagesToBuild | ||
|
||
if __name__ == "__main__": | ||
apps = sys.argv[1] | ||
forRelease = sys.argv[2] == "true" | ||
force = sys.argv[3] == "true" | ||
hash = sys.argv[4] | ||
imagesToBuild = { | ||
"images": [], | ||
"imagePlatforms": [] | ||
} | ||
|
||
if apps != "all": | ||
channels=None | ||
apps = apps.split(",") | ||
if len(sys.argv) == 5: | ||
channels = sys.argv[4].split(",") | ||
|
||
for app in apps: | ||
if not os.path.exists(os.path.join(CONTAINER_DIR, app)): | ||
print(f"App \"{app}\" not found") | ||
exit(1) | ||
|
||
meta = None | ||
if os.path.isfile(os.path.join(CONTAINER_DIR, app, "metadata.yaml")): | ||
meta = load_metadata_file_yaml(os.path.join(CONTAINER_DIR, app, "metadata.yaml")) | ||
elif os.path.isfile(os.path.join(CONTAINER_DIR, app, "metadata.json")): | ||
meta = load_metadata_file_json(os.path.join(CONTAINER_DIR, app, "metadata.json")) | ||
|
||
imageToBuild = get_image_metadata(os.path.join(CONTAINER_DIR, app), meta, hash, forRelease, force=force, channels=channels) | ||
if imageToBuild is not None: | ||
imagesToBuild["images"].extend(imageToBuild["images"]) | ||
imagesToBuild["imagePlatforms"].extend(imageToBuild["imagePlatforms"]) | ||
else: | ||
for subdir, dirs, files in os.walk(CONTAINER_DIR): | ||
for file in files: | ||
meta = None | ||
if file == "metadata.yaml": | ||
meta = load_metadata_file_yaml(os.path.join(subdir, file)) | ||
elif file == "metadata.json": | ||
meta = load_metadata_file_json(os.path.join(subdir, file)) | ||
else: | ||
continue | ||
if meta is not None: | ||
imageToBuild = get_image_metadata(subdir, meta, hash, forRelease, force=force) | ||
if imageToBuild is not None: | ||
imagesToBuild["images"].extend(imageToBuild["images"]) | ||
imagesToBuild["imagePlatforms"].extend(imageToBuild["imagePlatforms"]) | ||
print(json.dumps(imagesToBuild)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
import os | ||
import json | ||
import requests | ||
import yaml | ||
|
||
from jinja2 import Environment, PackageLoader, select_autoescape | ||
|
||
repo_owner = os.environ.get('REPO_OWNER', os.environ.get('GITHUB_REPOSITORY_OWNER')) | ||
repo_name = os.environ.get('REPO_NAME', os.environ.get('GITHUB_REPOSITORY')) | ||
|
||
env = Environment( | ||
loader=PackageLoader("render-readme"), | ||
autoescape=select_autoescape() | ||
) | ||
|
||
def load_metadata_file_yaml(file_path): | ||
with open(file_path, "r") as f: | ||
return yaml.safe_load(f) | ||
|
||
def load_metadata_file_json(file_path): | ||
with open(file_path, "r") as f: | ||
return json.load(f) | ||
|
||
def load_metadata_file(file_path): | ||
if file_path.endswith(".json"): | ||
return load_metadata_file_json(file_path) | ||
elif file_path.endswith(".yaml"): | ||
return load_metadata_file_yaml(file_path) | ||
return None | ||
|
||
if __name__ == "__main__": | ||
app_images = [] | ||
for subdir, dirs, files in os.walk("./containers"): | ||
for file in files: | ||
if file != "metadata.yaml" and file != "metadata.json": | ||
continue | ||
meta = load_metadata_file(os.path.join(subdir, file)) | ||
for channel in meta["channels"]: | ||
name = "" | ||
if channel.get("stable", False): | ||
name = meta["app"] | ||
else: | ||
name = "-".join([meta["app"], channel["name"]]) | ||
image = { | ||
"name": name, | ||
"channel": channel["name"], | ||
"html_url": f"https://github.com/{repo_name}/pkgs/container/{name}", | ||
"owner": repo_owner | ||
} | ||
|
||
app_images.append(image) | ||
|
||
template = env.get_template("README.md.j2") | ||
with open("./README.md", "w") as f: | ||
f.write(template.render(app_images=app_images)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
requests | ||
pyyaml | ||
packaging | ||
jinja2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
<!--- | ||
NOTE: AUTO-GENERATED FILE | ||
to edit this file, instead edit its template at: ./github/scripts/templates/README.md.j2 | ||
--> | ||
<div align="center"> | ||
## Containers | ||
</div> | ||
|
||
Welcome to gcsim container images. This registry contains containers for various gcsim services. | ||
|
||
## Tag immutability | ||
|
||
The containers built here do not use immutable tags, as least not in the more common way you have seen from [linuxserver.io](https://fleet.linuxserver.io/) or [Bitnami](https://bitnami.com/stacks/containers). | ||
|
||
We do take a similar approach but instead of appending a `-ls69` or `-r420` prefix to the tag we instead insist on pinning to the sha256 digest of the image, while this is not as pretty it is just as functional in making the images immutable. | ||
|
||
| Container | Immutable | | ||
|----------------------------------------------------|-----------| | ||
| `ghcr.io/genshinsim/embedgenerator:rolling` | ❌ | | ||
| `ghcr.io/genshinsim/embedgenerator:3.0.8.1507` | ❌ | | ||
| `ghcr.io/genshinsim/embedgenerator:rolling@sha256:8053...` | ✅ | | ||
| `ghcr.io/genshinsim/embedgenerator:3.0.8.1507@sha256:8053...` | ✅ | | ||
|
||
_If pinning an image to the sha256 digest, tools like [Renovate](https://github.com/renovatebot/renovate) support updating the container on a digest or application version change._ | ||
|
||
|
||
## Available Images | ||
|
||
Each Image will be built with a `rolling` tag, along with tags specific to it's version. Available Images Below | ||
|
||
Container | Channel | Image | ||
--- | --- | --- | ||
{% for image in app_images | sort(attribute="name") -%} | ||
[{{ image.name }}]({{ image.html_url }}) | {{ image.channel }} | ghcr.io/{{ image.owner }}/{{ image.name }} | ||
{% endfor %} | ||
|
||
## Credits | ||
|
||
This build script is based on the hard work of [onedr0p](https://github.com/onedr0p) and [joryirving](https://github.com/joryirving). |
Oops, something went wrong.