Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [TKC-2623] testkube kind Docker image #5867

Merged
merged 12 commits into from
Oct 2, 2024
30 changes: 30 additions & 0 deletions build/kind/cloud/Dockerfile.kind
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# syntax=docker/dockerfile:1
# Step 1: Use a base image with Docker installed
FROM docker:20.10.24-dind

# Step 2: Install necessary dependencies (curl, bash, tini)
RUN apk add --no-cache bash curl tini

# Step 3: Install Kind (Kubernetes in Docker)
RUN curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64 && \
chmod +x ./kind && \
mv ./kind /usr/local/bin/kind

# Step 4: Install kubectl (for interacting with the Kubernetes cluster)
RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
chmod +x ./kubectl && \
mv ./kubectl /usr/local/bin/kubectl

# Step 5: Install Helm (package manager for Kubernetes)
RUN curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Step 6: Script to automatically create Kind cluster and install Testkube
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh

# Step 7: Example K6 Test Workkflow CRD and preload Kind images
COPY ./images /images
COPY k6.yaml /examples/k6.yaml

# Step 8: Set Docker entry point for DIND (Docker-in-Docker)
ENTRYPOINT ["tini", "--", "/usr/local/bin/entrypoint.sh"]
151 changes: 151 additions & 0 deletions build/kind/cloud/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
#!/bin/bash

# Turn on bash's job control
set -m

# Logging function to make it easier to debug
log() {
echo "[INFO] $1"
}

# Check if agent key is provided
if [ -z "$AGENT_KEY" ]; then
log "Please provide AGENT_KEY env var"
exit 1
fi

# Check if org id is provided
if [ -z "$ORG_ID" ]; then
log "Please provide ORG_ID env var"
exit 1
fi

# Check if env id is provided
if [ -z "$ENV_ID" ]; then
log "Please provide ENV_ID env var"
exit 1
fi

# Check if cloud url is provided
if [ -z "$CLOUD_URL" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that rather ROOT_DOMAIN?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just followed exsitng agent helm installation command --set testkube-api.cloud.url=agent.testkube.io:443, will ROOT_DOMAI will be clear for user?

log "Please provide CLOUD_URL env var"
exit 1
fi

# Step 1: Start docker service in background
/usr/local/bin/dockerd-entrypoint.sh &

# Step 2: Wait that the docker service is up
while ! docker info; do
log "Waiting docker for 5 seconds..."
sleep 5
done

# Step 3: Import pre-installed images
for file in /images/*.tar; do
log "Load docker image $file..."
docker load <$file
done

# Step 4: Create Kind cluster using a specific Kubernetes version
log "Creating Kubernetes cluster using Kind (Kubernetes v1.31.0)..."
kind create cluster --name testkube-cluster --image kindest/node:v1.31.0 --wait 5m
if [ $? -ne 0 ]; then
log "Failed to create Kind cluster."
exit 1
fi

# Step 5: Verify kubectl is connected to the cluster
log "Verifying cluster is up..."
kubectl cluster-info
if [ $? -ne 0 ]; then
log "Failed to verify cluster."
exit 1
fi

# Step 6: Add the Testkube Helm repository
log "Adding Testkube Helm repository..."
helm repo add testkube https://kubeshop.github.io/helm-charts
helm repo update

# Step 7: Install Testkube using Helm
log "Installing Testkube via Helm..."
helm install testkube testkube/testkube --namespace testkube --create-namespace --set testkube-api.cloud.key=$AGENT_KEY --set testkube-api.cloud.orgId=$ORG_ID --set testkube-api.cloud.envId=$ENV_ID --set testkube-api.minio.enabled=false --set mongodb.enabled=false --set testkube-dashboard.enabled=false --set testkube-api.cloud.url=$CLOUD_URL
if [ $? -ne 0 ]; then
log "Testkube installation failed."
exit 1
fi

# Step 8: Verify Testkube is installed to the cluster
log "Verifying Testkube is up..."
counter=0
log_pattern="starting Testkube API server"
while [ $counter -lt 15 ]
do
# Get all pod statuses in the Testkube namespace
pod_status=$(kubectl get pods -n testkube --no-headers)

# Check if there are any pods in the Testkube namespace
if [ -z "$pod_status" ]; then
log "No pods found in Testkube namespace."
exit 1
fi

# Iterate through each pod, check status and log pattern
all_running=true
found_pattern=false

log "Checking pods in Testkube namespace..."
while read -r line; do
pod_name=$(echo "$line" | awk '{print $1}')
status=$(echo "$line" | awk '{print $3}')

if [ "$status" != "Running" ]; then
log "Pod $pod_name is not running. Status: $status."
all_running=false
break
else
log "Pod $pod_name is running."
fi

if [[ $pod_name == *"testkube-api-server"* ]]; then
pod_logs=$(kubectl logs "$pod_name" -n testkube)

# Check if logs contain the desired pattern
if echo "$pod_logs" | grep -q "$log_pattern"; then
log "Log pattern found: $log_pattern."
found_pattern=true
else
log "Log pattern not found: $log_pattern."
break
fi
fi
done <<< "$pod_status"

if [ "$all_running" = true ] && [ "$found_pattern" = true ] ; then
log "Waiting Testkube API for 30 seconds..."
sleep 30
break
else
log "Waiting Testkube for 30 seconds..."
sleep 30
fi

counter=$(( counter + 1 ))
done

if [ $counter -eq 15 ]; then
log "Testkube validation failed."
exit 1
fi
log "Testkube is up and running."

# Step 9: Create Testkube k6 Test Workflow
log "Creating and running Testkube k6 Test Workflow..."
kubectl apply -f /examples/k6.yaml -n testkube

log "Testkube installation successful!"
log "You can now use Testkube in your Kind Kubernetes cluster."

# Step 10: Bring docker service back to foreground
fg %1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it maybe instead stream Agent + Control Plane logs?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just for Agent for now. actually, last part is for demoing purpose, it's not needed for agent work. so user can see that it works. he can always run the shell and check the agent logs

Loading
Loading