openshift 4.9 single node, assisted install mode, without dhcp, connected

本文描述，如何使用assisted service(辅助安装服务)，来安装一个单节点openshift4集群，特别的地方是，默认情况，openshift4要求网络上提供dhcp服务，让节点启动的时候，能拿到IP地址，从而进一步下载容器镜像，并且和assisted service交互，拿到配置。可是大部分客户的网络，是不允许开启dhcp服务的，那么我们在这里就使用assisted service暂时隐藏的功能，进行static ip模式的部署。

本实验设想的客户环境/需求是这样的：

实验网络没有dhcp
实验网络可以访问外网
实验环境中有2台主机
将在实验环境中的1台主机上，安装单节点openshift4(baremetal模式)

由于作者实验环境所限，我们就用kvm来代替baremetal进行实验。

安装过程大概是这样的：

启动helper vm，并在helper节点上配置dns服务
启动本地assisted service服务
在assisted service上进行配置
从assisted service上下载iso
通过iso启动kvm/baremetal
在assisted service上进行配置，开始安装
观察和等待安装结束
获得openshift4的用户名密码等信息，登录集群。

本次实验的架构图：

部署 dns

assisted install 模式下，如果想静态ip安装，需要在实验网络上部署一个dns服务。因为我们部署的是single node openshift，只需要把如下4个域名，指向同一个ip地址就可以。当然，你需要提前想好域名。

api.ocp4s.redhat.ren
api-int.ocp4s.redhat.ren
*.apps.ocp4.redhat.ren
ocp4-sno.ocp4.redhat.ren

部署 assisted install service

assisted install service有2个版本，一个是cloud.redhat.com上面那个，同时还有一个本地版本，两个版本功能一样，因为我们需要有定制需求，所以我们选择本地版本。

# https://github.com/openshift/assisted-service/blob/master/docs/user-guide/assisted-service-on-local.md

# https://github.com/openshift/assisted-service/tree/master/deploy/podman

podman version
# Version:      3.4.2
# API Version:  3.4.2
# Go Version:   go1.16.12
# Built:        Wed Feb  2 07:59:28 2022
# OS/Arch:      linux/amd64

mkdir -p /data/assisted-service/
cd /data/assisted-service/

export http_proxy="http://192.168.195.54:5085"
export https_proxy=${http_proxy}

wget https://raw.githubusercontent.com/openshift/assisted-service/master/deploy/podman/configmap.yml
wget https://raw.githubusercontent.com/openshift/assisted-service/master/deploy/podman/pod.yml

unset http_proxy
unset https_proxy

sed -i 's/ SERVICE_BASE_URL:.*/ SERVICE_BASE_URL: "http:\/\/172.21.6.103:8090"/' configmap.yml

# 启动本地assisted service
podman play kube --configmap configmap.yml pod.yml

# 用以下命令，停止/删除本地assisted service
cd /data/assisted-service/
podman play kube --down pod.yml

⚠️注意：本地版本的assisted service，会从mirror.openshift.com上面下载多个版本的iso，总共有6GB。请等待下载完成

podman exec assisted-installer-image-service du -h /data
# 6.3G    /data

运行成功以后，访问以下url

http://172.21.6.103:8080

创建cluster

访问本地的assist install service, 创建一个cluster

配置集群的基本信息

填写自己的pull-secret信息，并点击下一步

进入下一个页面后，点击add host

直接点击generate discovery iso，我们会在后面定制ssh key，现在不需要配置。

记录下来download command，因为我们需要里面的env infra id

我们这里的command是

wget -O discovery_image_ocp4s.iso 'http://127.0.0.1:8888/images/78506b3c-46e4-47f7-8a18-ec1ca4baa3b9?arch=x86_64&type=full-iso&version=4.9'

定制 assisted install service的配置

assisted install service创建的iso，要去实验网络必须有dhcp服务，我们要做的是static ip，那么我们就要定制一下 assisted install service, 激活他现在还是隐藏的功能（暂时没有官方支持）。

# on helper
cd /data/sno

SNO_IP=172.21.6.13
SNO_GW=172.21.6.254
SNO_NETMAST=255.255.255.0
SNO_NETMAST_S=24
SNO_HOSTNAME=ocp4-sno
SNO_IF=enp1s0
SNO_IF_MAC=`printf '00:60:2F:%02X:%02X:%02X' $[RANDOM%256] $[RANDOM%256] $[RANDOM%256]`
SNO_DNS=172.21.1.1
SNO_DISK=/dev/vda
SNO_CORE_PWD=redhat

echo ${SNO_IF_MAC} > /data/sno/sno.mac

ASSISTED_SERVICE_URL=http://172.21.6.103:8080
# infra id is part of download url on UI
INFRA_ENV_ID=78506b3c-46e4-47f7-8a18-ec1ca4baa3b9
NODE_SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
request_body=$(mktemp)

cat << EOF > /data/sno/server-a.yaml
dns-resolver:
  config:
    server:
    - ${SNO_DNS}
interfaces:
- ipv4:
    address:
    - ip: ${SNO_IP}
      prefix-length: ${SNO_NETMAST_S}
    dhcp: false
    enabled: true
  name: ${SNO_IF}
  state: up
  type: ethernet
routes:
  config:
  - destination: 0.0.0.0/0
    next-hop-address: ${SNO_GW}
    next-hop-interface: ${SNO_IF}
    table-id: 254
EOF

cat << EOF > /data/sno/static.ip.bu
variant: openshift
version: 4.9.0
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-zzz-master-static-ip
storage:
  files:
    - path: /etc/NetworkManager/system-connections/${SNO_IF}.nmconnection
      overwrite: true
      contents:
        inline: |
          [connection]
          id=${SNO_IF}
          type=ethernet
          autoconnect-retries=1
          interface-name=${SNO_IF}
          multi-connect=1
          permissions=
          wait-device-timeout=60000

          [ethernet]
          mac-address-blacklist=

          [ipv4]
          address1=${SNO_IP}/${SNO_NETMAST_S=24},${SNO_GW}
          dhcp-hostname=${SNO_HOSTNAME}
          dhcp-timeout=90
          dns=${SNO_DNS};
          dns-search=
          may-fail=false
          method=manual

          [ipv6]
          addr-gen-mode=eui64
          dhcp-hostname=${SNO_HOSTNAME}
          dhcp-timeout=90
          dns-search=
          method=disabled

          [proxy]

EOF

# https://access.redhat.com/solutions/6194821
# butane /data/sno/static.ip.bu | python3 -c 'import json, yaml, sys; print(json.dumps(yaml.load(sys.stdin)))'


# https://stackoverflow.com/questions/2854655/command-to-escape-a-string-in-bash
# VAR_PULL_SEC=`printf "%q" $(cat  /data/pull-secret.json)`
tmppath=$(mktemp)
butane /data/sno/static.ip.bu | python3 -c 'import json, yaml, sys; print(json.dumps(yaml.load(sys.stdin)))' | jq -c '.spec.config | .ignition.version = "3.1.0" ' > ${tmppath}
VAR_NMSTATIC=$(cat ${tmppath})
# rm -f ${tmppath}

jq -n --arg SSH_KEY "$NODE_SSH_KEY" \
  --arg NMSTATE_YAML1 "$(cat server-a.yaml)" \
  --arg MAC_ADDR "$(cat /data/sno/sno.mac)" \
  --arg PULL_SEC "$(cat  /data/pull-secret.json)" \
  --arg NMSTATIC "${VAR_NMSTATIC}" \
'{
    "proxy":{"http_proxy":"","https_proxy":"","no_proxy":""},
    "ssh_authorized_key":$SSH_KEY,
    "pull_secret":$PULL_SEC,
    "image_type":"full-iso",
    "ignition_config_override":$NMSTATIC,
    "static_network_config": [
      {
        "network_yaml": $NMSTATE_YAML1,
        "mac_interface_map": [{"mac_address": $MAC_ADDR, "logical_nic_name": "enp1s0"}]
      }
    ]
}' > $request_body

# 我们来看看创建的request body
cat $request_body

# 向 assisted install service发送请求，进行定制
curl -H "Content-Type: application/json" -X PATCH -d @$request_body ${ASSISTED_SERVICE_URL}/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID
# {"cluster_id":"850934fd-fa64-4057-b9d2-1eeebd890e1a","cpu_architecture":"x86_64","created_at":"2022-02-11T03:54:46.632598Z","download_url":"http://127.0.0.1:8888/images/89cc84a1-2dfd-4d7e-9ca3-903342c40d60?arch=x86_64&type=full-iso&version=4.9","email_domain":"Unknown","expires_at":"0001-01-01T00:00:00.000Z","href":"/api/assisted-install/v2/infra-envs/89cc84a1-2dfd-4d7e-9ca3-903342c40d60","id":"89cc84a1-2dfd-4d7e-9ca3-903342c40d60","kind":"InfraEnv","name":"ocp4s_infra-env","openshift_version":"4.9","proxy":{"http_proxy":"","https_proxy":"","no_proxy":""},"pull_secret_set":true,"ssh_authorized_key":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrkO4oLIFTwjkGON+aShlQRKwXHOf3XKrGDmpb+tQM3UcbsF2U7klsr9jBcGObQMZO7KBW8mlRu0wC2RxueBgjbqvylKoFacgVZg6PORfkclqE1gZRYFwoxDkLo2c5y5B7OhcAdlHO0eR5hZ3/0+8ZHZle0W+A0AD7qqowO2HlWLkMMt1QXFD7R0r6dzTs9u21jASGk3jjYgCOw5iHvqm2ueVDFAc4yVwNZ4MXKg5MRvqAJDYPqhaRozLE60EGIziy9SRj9HWynyNDncCdL1/IBK2z9T0JwDebD6TDNcPCtL+AeKIpaHed52PkjnFf+Q+8/0Z0iXt6GyFYlx8OkxdsiMgMxiXx43yIRaWZjx54kVtc9pB6CL50UKPQ2LjuFPIZSfaCab5KDgPRtzue82DE6Mxxg4PS+FTW32/bq1WiOxCg9ABrZ0n1CGaZWFepJkSw47wodMnvlBkcKY3Rn/SsLZVOUsJysd+b08LQgl1Fr3hjVrEQMLbyU0UxvoerYfk= root@ocp4-helper","static_network_config":"dns-resolver:\n  config:\n    server:\n    - 172.21.1.1\ninterfaces:\n- ipv4:\n    address:\n    - ip: 172.21.6.13\n      prefix-length: 24\n    dhcp: false\n    enabled: true\n  name: enp1s0\n  state: up\n  type: ethernet\nroutes:\n  config:\n  - destination: 0.0.0.0/0\n    next-hop-address: 172.21.6.254\n    next-hop-interface: enp1s0\n    table-id: 254HHHHH00:60:2F:8B:42:88=enp1s0","type":"full-iso","updated_at":"2022-02-11T04:01:14.008388Z","user_name":"admin"}

# on helper
cd /data/sno/
wget -O discovery_image_ocp4s.iso "http://172.21.6.103:8888/images/${INFRA_ENV_ID}?arch=x86_64&type=full-iso&version=4.9"

# coreos-installer iso kargs modify -a \
#   " ip=${SNO_IP}::${SNO_GW}:${SNO_NETMAST}:${SNO_HOSTNAME}:${SNO_IF}:none nameserver=${SNO_DNS}" \
#   /data/sno/discovery_image_ocp4s.iso

/bin/mv -f /data/sno/discovery_image_ocp4s.iso /data/sno/sno.iso

启动kvm

我们回到kvm宿主机，启动kvm，开始安装single node openshift

# back to kvm host

create_lv() {
    var_vg=$1
    var_lv=$2
    var_size=$3
    lvremove -f $var_vg/$var_lv
    lvcreate -y -L $var_size -n $var_lv $var_vg
    wipefs --all --force /dev/$var_vg/$var_lv
}

create_lv vgdata lvsno 120G

export KVM_DIRECTORY=/data/kvm

mkdir -p  ${KVM_DIRECTORY}
cd ${KVM_DIRECTORY}
scp root@192.168.7.11:/data/sno/sno.* ${KVM_DIRECTORY}/

# on kvm host
# export KVM_DIRECTORY=/data/kvm
virt-install --name=ocp4-sno --vcpus=16 --ram=65536 \
--cpu=host-model \
--disk path=/dev/vgdata/lvsno,device=disk,bus=virtio,format=raw \
--os-variant rhel8.3 --network bridge=baremetal,model=virtio,mac=$(<sno.mac) \
--graphics vnc,port=59012 \
--boot menu=on --cdrom ${KVM_DIRECTORY}/sno.iso

在 assisted install service里面配置sno参数

回到 assisted install service webUI，能看到node已经被发现

点击下一步，配置物理机的安装子网

点击下一步，回顾集群配置信息

开始安装，到这里，我们等待就可以

一段时间以后，通常20-30分钟，就安装完成了，当然这要网络情况比较好的条件下。

⚠️不要忘记下载集群证书，还有webUI的用户名，密码。

访问sno集群

# back to helper
# copy kubeconfig from web browser to /data/sno
export KUBECONFIG=/data/sno/auth/kubeconfig

oc get node
# NAME       STATUS   ROLES           AGE   VERSION
# ocp4-sno   Ready    master,worker   71m   v1.22.3+e790d7f

oc get co
# NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
# authentication                             4.9.18    True        False         False      54m
# baremetal                                  4.9.18    True        False         False      58m
# cloud-controller-manager                   4.9.18    True        False         False      63m
# cloud-credential                           4.9.18    True        False         False      68m
# cluster-autoscaler                         4.9.18    True        False         False      59m
# config-operator                            4.9.18    True        False         False      69m
# console                                    4.9.18    True        False         False      54m
# csi-snapshot-controller                    4.9.18    True        False         False      68m
# dns                                        4.9.18    True        False         False      58m
# etcd                                       4.9.18    True        False         False      62m
# image-registry                             4.9.18    True        False         False      55m
# ingress                                    4.9.18    True        False         False      57m
# insights                                   4.9.18    True        False         False      63m
# kube-apiserver                             4.9.18    True        False         False      58m
# kube-controller-manager                    4.9.18    True        False         False      61m
# kube-scheduler                             4.9.18    True        False         False      60m
# kube-storage-version-migrator              4.9.18    True        False         False      68m
# machine-api                                4.9.18    True        False         False      59m
# machine-approver                           4.9.18    True        False         False      60m
# machine-config                             4.9.18    True        False         False      63m
# marketplace                                4.9.18    True        False         False      68m
# monitoring                                 4.9.18    True        False         False      54m
# network                                    4.9.18    True        False         False      68m
# node-tuning                                4.9.18    True        False         False      64m
# openshift-apiserver                        4.9.18    True        False         False      55m
# openshift-controller-manager               4.9.18    True        False         False      60m
# openshift-samples                          4.9.18    True        False         False      57m
# operator-lifecycle-manager                 4.9.18    True        False         False      60m
# operator-lifecycle-manager-catalog         4.9.18    True        False         False      60m
# operator-lifecycle-manager-packageserver   4.9.18    True        False         False      58m
# service-ca                                 4.9.18    True        False         False      68m
# storage                                    4.9.18    True        False         False      63m

访问集群的webUI

https://console-openshift-console.apps.ocp4s.redhat.ren/

用户名密码是： kubeadmin / 3QS3M-HA3Px-376HD-bvfif

reference

https://github.com/openshift/assisted-service/tree/master/docs/user-guide

https://access.redhat.com/solutions/6135171
https://github.com/openshift/assisted-service/blob/master/docs/user-guide/assisted-service-on-local.md
https://github.com/openshift/assisted-service/blob/master/docs/user-guide/restful-api-guide.md

search

pre-network-manager-config.sh
/Users/wzh/Desktop/dev/assisted-service/internal/constants/scripts.go
NetworkManager

https://superuser.com/questions/218340/how-to-generate-a-valid-random-mac-address-with-bash-shell

end

cat << EOF > test
02:00:00:2c:23:a5=enp1s0
EOF
cat test | cut -d= -f1 | tr '[:lower:]' '[:upper:]'

printf '00-60-2F-%02X-%02X-%02X\n' $[RANDOM%256] $[RANDOM%256] $[RANDOM%256]
virsh domifaddr freebsd11.1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.9.sno.static.ip.local.assisted.connected.md

4.9.sno.static.ip.local.assisted.connected.md

openshift 4.9 single node, assisted install mode, without dhcp, connected

部署 dns

部署 assisted install service

创建cluster

定制 assisted install service的配置

启动kvm

在 assisted install service里面配置sno参数

访问sno集群

reference

end

Files

4.9.sno.static.ip.local.assisted.connected.md

Latest commit

History

4.9.sno.static.ip.local.assisted.connected.md

File metadata and controls

openshift 4.9 single node, assisted install mode, without dhcp, connected

部署 dns

部署 assisted install service

创建cluster

定制 assisted install service的配置

启动kvm

在 assisted install service里面配置sno参数

访问sno集群

reference

end