Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Nightly Test runs to pass #701

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Improve Nightly Test runs to pass #701

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ade18f4
Use the correct snap when upgrading
addyess Sep 27, 2024
29c0ec0
Make etcd arch aware
addyess Sep 27, 2024
8af94cc
bump etcd version to 3.4.34
addyess Sep 27, 2024
86176e8
Only use tmate if we're debugging a pull-reqeust
addyess Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 24 additions & 13 deletions .github/workflows/nightly-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,47 +3,43 @@ name: Nightly Latest/Edge Tests
on:
schedule:
- cron: '0 0 * * *' # Runs every midnight
pull_request:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 Don't merge with this in there -- its just for testing this PR 🚨


permissions:
contents: read

jobs:
test-integration:
name: Integration Test ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.releases }}
name: Integration Test ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.release }}
strategy:
matrix:
os: ["ubuntu:20.04", "ubuntu:22.04", "ubuntu:24.04"]
arch: ["amd64", "arm64"]
releases: ["latest/edge"]
release: ["latest/edge"]
fail-fast: false # TODO: remove once arm64 works

runs-on: ${{ matrix.arch == 'arm64' && 'Ubuntu_ARM64_4C_16G_01' || 'ubuntu-20.04' }}

steps:
- name: Checking out repo
uses: actions/checkout@v4
- name: Setup Python
- name: Install lxd and tox
run: |
sudo apt update
sudo apt install -y python3 python3-pip
- name: Install tox
run: |
pip3 install tox==4.13
- name: Install lxd
run: |
sudo apt install -y tox
sudo snap refresh lxd --channel 5.21/stable
sudo lxd init --auto
sudo usermod --append --groups lxd $USER
sg lxd -c 'lxc version'
- name: Create build directory
run: mkdir -p build
- name: Install $${ matrix.releases }} k8s snap
- name: Install ${{ matrix.release }} k8s snap
run: |
cd build
snap download k8s --channel=${{ matrix.releases }} --basename k8s
snap download k8s --channel=${{ matrix.release }} --basename k8s
- name: Run end to end tests # tox path needs to be specified for arm64
env:
TEST_SNAP: ${{ github.workspace }}/build/k8s-${{ matrix.patch }}.snap
TEST_SNAP: ${{ github.workspace }}/build/k8s.snap
TEST_SUBSTRATE: lxd
TEST_LXD_IMAGE: ${{ matrix.os }}
TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports
Expand All @@ -52,4 +48,19 @@ jobs:
TEST_VERSION_UPGRADE_CHANNELS: "recent 6 classic"
run: |
export PATH="/home/runner/.local/bin:$PATH"
cd tests/integration && sg lxd -c 'tox -e integration'
cd tests/integration && sg lxd -c 'tox -vve integration'
- name: Prepare inspection reports
if: failure()
run: |
tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports
echo "artifact_name=inspection-reports-${{ matrix.os }}-${{ matrix.arch }}" | sed 's/:/-/g' >> $GITHUB_ENV
- name: Upload inspection report artifact
if: failure()
uses: actions/upload-artifact@v4
with:
name: ${{ env.artifact_name }}
path: ${{ github.workspace }}/inspection-reports.tar.gz
- name: Tmate debugging session
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO this will just extend the failed test for 10min. Nobody is looking at the nightly tests until they fail. The inspection report should be sufficient to reproduce the issue, if not, we need to isolate the failed test and add the tmate session as part of a draft/test PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, we can take it out. I had it in there b/c i was tinkering and trying to learn why things failed but weren't in the inspection reports

if: ${{ failure() && github.event_name == 'pull_request' }}
uses: mxschmitt/action-tmate@v3
timeout-minutes: 10
1 change: 1 addition & 0 deletions tests/integration/templates/etcd/etcd.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
LimitNOFILE=40000
TimeoutStartSec=0

Environment=ETCD_UNSUPPORTED_ARCH=$ARCH
addyess marked this conversation as resolved.
Show resolved Hide resolved
ExecStart=/tmp/test-etcd/etcd --name $NAME \
--data-dir /tmp/etcd/s1 \
--listen-client-urls $CLIENT_URL \
Expand Down
2 changes: 1 addition & 1 deletion tests/integration/tests/test_util/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
ETCD_URL = os.getenv("ETCD_URL") or "https://github.com/etcd-io/etcd/releases/download"

# ETCD_VERSION is the version of etcd to use.
ETCD_VERSION = os.getenv("ETCD_VERSION") or "v3.3.8"
ETCD_VERSION = os.getenv("ETCD_VERSION") or "v3.4.34"

# SNAP is the absolute path to the snap against which we run the integration tests.
SNAP = os.getenv("TEST_SNAP")
Expand Down
8 changes: 5 additions & 3 deletions tests/integration/tests/test_util/etcd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ def add_node(self):
]

substitutes = {
"ARCH": instance.arch,
"NAME": instance.id,
"IP": ip,
"CLIENT_URL": f"https://{ip}:2379",
Expand Down Expand Up @@ -224,21 +225,22 @@ def add_node(self):
input=str.encode(src.substitute(substitutes)),
)

arch = instance.arch
instance.exec(
[
"curl",
"-L",
f"{self.etcd_url}/{self.etcd_version}/etcd-{self.etcd_version}-linux-amd64.tar.gz",
f"{self.etcd_url}/{self.etcd_version}/etcd-{self.etcd_version}-linux-{arch}.tar.gz",
"-o",
f"/tmp/etcd-{self.etcd_version}-linux-amd64.tar.gz",
f"/tmp/etcd-{self.etcd_version}-linux-{arch}.tar.gz",
]
)
instance.exec(["mkdir", "-p", "/tmp/test-etcd"])
instance.exec(
[
"tar",
"xzvf",
f"/tmp/etcd-{self.etcd_version}-linux-amd64.tar.gz",
f"/tmp/etcd-{self.etcd_version}-linux-{arch}.tar.gz",
"-C",
"/tmp/test-etcd",
"--strip-components=1",
Expand Down
8 changes: 7 additions & 1 deletion tests/integration/tests/test_util/harness/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Copyright 2024 Canonical, Ltd.
#
import subprocess
from functools import partial
from functools import cached_property, partial


class HarnessError(Exception):
Expand Down Expand Up @@ -30,6 +30,12 @@ def __init__(self, h: "Harness", id: str) -> None:
def id(self) -> str:
return self._id

@cached_property
def arch(self) -> str:
return self.exec(
["dpkg", "--print-architecture"], text=True, capture_output=True
).stdout.strip()

Comment on lines +33 to +38
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay! an instance can know what architecture its running on

def __str__(self) -> str:
return f"{self._h.name}:{self.id}"

Expand Down
7 changes: 2 additions & 5 deletions tests/integration/tests/test_version_upgrades.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

@pytest.mark.node_count(1)
@pytest.mark.no_setup()
@pytest.mark.xfail("cilium failures are blocking this from working")
@pytest.mark.xfail(reason="cilium failures are blocking this from working")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

repairs a syntax error on this xfail call

@pytest.mark.skipif(
not config.VERSION_UPGRADE_CHANNELS, reason="No upgrade channels configured"
)
Expand All @@ -26,10 +26,7 @@ def test_version_upgrades(instances: List[harness.Instance]):
"'recent' requires the number of releases as second argument and the flavour as third argument"
)
_, num_channels, flavour = channels
arch = cp.exec(
["dpkg", "--print-architecture"], text=True, capture_output=True
).stdout.strip()
channels = snap.get_latest_channels(int(num_channels), flavour, arch)
channels = snap.get_latest_channels(int(num_channels), flavour, cp.arch)
addyess marked this conversation as resolved.
Show resolved Hide resolved

LOG.info(
f"Bootstrap node on {channels[0]} and upgrade through channels: {channels[1:]}"
Expand Down
33 changes: 16 additions & 17 deletions tests/integration/tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,47 +1,46 @@
[tox]
no_package = True
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these changes are to make this more compatible with tox 3. Why tox 3? Because that's what you get with apt on focal and jammy. Why not pip install it? Arm64 runners were having severe trouble loading python libraries from a virtual env on the GH runners for reasons i never was able to understand. I think it's a combination of running tox with sg but i'm not sure. LD_PYTHON_PATH got lost though -- using the system-wide tox improves this

skipsdist = True
skip_missing_interpreters = True
env_list = format, lint, integration
min_version = 4.0.0

[testenv]
set_env =
PYTHONBREAKPOINT=pdb.set_trace
PY_COLORS=1
pass_env =
passenv =
PYTHONPATH

[testenv:format]
description = Apply coding style standards to code
deps = -r {tox_root}/requirements-dev.txt
deps = -r {toxinidir}/requirements-dev.txt
commands =
licenseheaders -t {tox_root}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {tox_root}/tests
isort {tox_root}/tests --profile=black
black {tox_root}/tests
licenseheaders -t {toxinidir}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {toxinidir}/tests
isort {toxinidir}/tests --profile=black
black {toxinidir}/tests

[testenv:lint]
description = Check code against coding style standards
deps = -r {tox_root}/requirements-dev.txt
deps = -r {toxinidir}/requirements-dev.txt
commands =
codespell {tox_root}/tests
flake8 {tox_root}/tests
licenseheaders -t {tox_root}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {tox_root}/tests --dry
isort {tox_root}/tests --profile=black --check
black {tox_root}/tests --check --diff
codespell {toxinidir}/tests
flake8 {toxinidir}/tests
licenseheaders -t {toxinidir}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {toxinidir}/tests --dry
isort {toxinidir}/tests --profile=black --check
black {toxinidir}/tests --check --diff

[testenv:integration]
description = Run integration tests
deps =
-r {tox_root}/requirements-test.txt
-r {toxinidir}/requirements-test.txt
commands =
pytest -v \
pytest -vv \
--maxfail 1 \
--tb native \
--log-cli-level DEBUG \
--disable-warnings \
{posargs} \
{tox_root}/tests
pass_env =
{toxinidir}/tests
passenv =
TEST_*

[flake8]
Expand Down
Loading