介绍
麻雀虽小，五脏俱全。MicroK8s 是 CNCF 认证的 Kubernetes 发行版，由 Ubuntu 背后的商业公司 Canonical 开发和维护。它和完整版的 Kubernetes 一样支持高可用特性（HA），支持快速组建 K8s 集群。适合用于边缘计算、IoT、以及使用 KubeFlow 的 MLOps 机器学习场景。也适合用于开发者本地环境，以轻量的资源消耗、简单的运维成本获得几乎完整的 Kubernetes 生态体验。
MicroK8s只需要有三台以上的主节点即可开启高可用特性。在 云平台-microk8s（一）：单机部署 中已经部署了一个单机的 microk8s ，现在只需要增加两台即可。
准备
| 主机 | 系统 | ip | cpu | 内存 | 硬盘 | 备注 |
|---|---|---|---|---|---|---|
| microk8s-02 | ubuntu 22.04 server | 192.168.1.249 | 2 | 4G | 40G | 新增 |
| microk8s-03 | ubuntu 22.04 server | 192.168.1.250 | 2 | 4G | 40G | 新增 |
同样使用 KVM 克隆出需要的虚拟机，直接克隆之前创建的虚拟机 microk8s-00 为 microk8s-02 microk8s-03 ，也可以使用其他方式获得。
克隆虚拟机
1
2
3
4
5
6
7
8
9
10
# 关闭虚拟机（克隆前必须先关闭）
virsh shutdown microk8s-00
# 克隆虚拟机 microk8s-02 microk8s-03
virt-clone -o microk8s-00 -n microk8s-02 -f /kvm/vm/microk8s-02.qcow2
virt-clone -o microk8s-00 -n microk8s-03 -f /kvm/vm/microk8s-03.qcow2
# 启动新虚拟机 microk8s-02 microk8s-03
virsh start microk8s-02
virsh start microk8s-03
修改虚拟机
这里不再累述，修改过程和 云平台-microk8s（一）：单机部署 中 microk8s-01 的修改过程一样，包括修改主机名、hosts、网络，并重启
注: 注意ip对应
192.168.1.248 microk8s-01
192.168.1.249 microk8s-02
192.168.1.250 microk8s-03
安装及问题解决
还是要通过命令安装 microk8s，并且同样会遇到问题。解决办法和 microk8s-01 一样，这里不再累述。
高可用
开启高可用前，先看看microk8s的状态
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
microk8s@microk8s-01:~$ microk8s.kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
container-registry pod/registry-6c9fcc695f-559lh 1/1 Running 0 6m49s
kube-system pod/calico-kube-controllers-77bd7c5b-9cqg6 1/1 Running 0 15h
kube-system pod/calico-node-sm97h 1/1 Running 0 15h
kube-system pod/coredns-864597b5fd-lhlfk 1/1 Running 0 15h
kube-system pod/dashboard-metrics-scraper-5657497c4c-5kt94 1/1 Running 0 11h
kube-system pod/hostpath-provisioner-756cd956bc-jv4jr 1/1 Running 0 6m51s
kube-system pod/kubernetes-dashboard-54b48fbf9-qtj4s 1/1 Running 0 11h
kube-system pod/metrics-server-848968bdcd-8lqff 1/1 Running 0 11h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
container-registry service/registry NodePort 10.152.183.79 <none> 5000:32000/TCP 6m49s
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 15h
kube-system service/dashboard-metrics-scraper ClusterIP 10.152.183.98 <none> 8000/TCP 11h
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 15h
kube-system service/kubernetes-dashboard ClusterIP 10.152.183.106 <none> 443/TCP 11h
kube-system service/metrics-server ClusterIP 10.152.183.215 <none> 443/TCP 11h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 15h
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
container-registry deployment.apps/registry 1/1 1 1 6m49s
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 15h
kube-system deployment.apps/coredns 1/1 1 1 15h
kube-system deployment.apps/dashboard-metrics-scraper 1/1 1 1 11h
kube-system deployment.apps/hostpath-provisioner 1/1 1 1 6m52s
kube-system deployment.apps/kubernetes-dashboard 1/1 1 1 11h
kube-system deployment.apps/metrics-server 1/1 1 1 11h
NAMESPACE NAME DESIRED CURRENT READY AGE
container-registry replicaset.apps/registry-6c9fcc695f 1 1 1 6m49s
kube-system replicaset.apps/calico-kube-controllers-77bd7c5b 1 1 1 15h
kube-system replicaset.apps/coredns-864597b5fd 1 1 1 15h
kube-system replicaset.apps/dashboard-metrics-scraper-5657497c4c 1 1 1 11h
kube-system replicaset.apps/hostpath-provisioner-756cd956bc 1 1 1 6m52s
kube-system replicaset.apps/kubernetes-dashboard-54b48fbf9 1 1 1 11h
kube-system replicaset.apps/metrics-server-848968bdcd 1 1 1 11h
microk8s@microk8s-01:~$ microk8s status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
metrics-server # (core) K8s Metrics Server for API access to service metrics
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
gpu # (core) Alias to nvidia add-on
host-access # (core) Allow Pods connecting to Host services smoothly
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
rook-ceph # (core) Distributed Ceph storage using Rook
可以看到，现在并没有开启高可用。之后我们通过简单的命令来开启高可用。
加入集群
1
2
3
4
5
6
7
8
9
10
11
# 在microk8s-01 上执行
root@microk8s-01:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.248:25000/f8cf3580ed1ac99294dad461bd8c322a/a28774bf36f6
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.1.248:25000/f8cf3580ed1ac99294dad461bd8c322a/a28774bf36f6 --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.248:25000/f8cf3580ed1ac99294dad461bd8c322a/a28774bf36f6
microk8s join 172.17.0.1:25000/f8cf3580ed1ac99294dad461bd8c322a/a28774bf36f6
注:
- 执行
microk8s add-node命令后会有三个提示和四条microk8s join命令，第一条没有--worker参数，第二条有--worker参数。第三条和第四条是不同的网卡ip。因为我们要开启高可用，这里使用第一条。- 有的版本好像没有
--worker参数的命令- 这个命令是一次有效的，如果再添加新的节点需要重新执行获取新的命令
1
2
3
4
5
6
# microk8s-02 上执行
root@microk8s-02:~# microk8s join 192.168.1.248:25000/f8cf3580ed1ac99294dad461bd8c322a/a28774bf36f6
Contacting cluster at 192.168.1.248
Waiting for this node to finish joining the cluster. .. .. .. .
Successfully joined the cluster.
root@microk8s-02:~#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
# microk8s-02 查看状态
root@microk8s-02:~# microk8s status
microk8s is running
high-availability: no
datastore master nodes: 192.168.1.248:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
metrics-server # (core) K8s Metrics Server for API access to service metrics
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
gpu # (core) Alias to nvidia add-on
host-access # (core) Allow Pods connecting to Host services smoothly
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
rook-ceph # (core) Distributed Ceph storage using Rook
root@microk8s-02:~#
# microk8s-01 查看状态
root@microk8s-01:~# microk8s status
microk8s is running
high-availability: no
datastore master nodes: 192.168.1.248:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
metrics-server # (core) K8s Metrics Server for API access to service metrics
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
gpu # (core) Alias to nvidia add-on
host-access # (core) Allow Pods connecting to Host services smoothly
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
rook-ceph # (core) Distributed Ceph storage using Rook
root@microk8s-01:~#
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# 在 microk8s-01 上重新获取命令
root@microk8s-01:~# microk8s add
add-node addons
root@microk8s-01:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.248:25000/2eecae7a7c45790ddee82ac9ffcd0ee2/a28774bf36f6
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.1.248:25000/2eecae7a7c45790ddee82ac9ffcd0ee2/a28774bf36f6 --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.248:25000/2eecae7a7c45790ddee82ac9ffcd0ee2/a28774bf36f6
microk8s join 172.17.0.1:25000/2eecae7a7c45790ddee82ac9ffcd0ee2/a28774bf36f6
root@microk8s-01:~#
# 在 microk8s-03 执行加入
root@microk8s-03:~# microk8s join 192.168.1.248:25000/2eecae7a7c45790ddee82ac9ffcd0ee2/a28774bf36f6
Contacting cluster at 192.168.1.248
Waiting for this node to finish joining the cluster. .. .. .. .
Successfully joined the cluster.
root@microk8s-03:~#
# 查看状态
root@microk8s-01:~# microk8s status
microk8s is running
high-availability: yes
datastore master nodes: 192.168.1.248:19001 192.168.1.249:19001 192.168.1.250:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
metrics-server # (core) K8s Metrics Server for API access to service metrics
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
gpu # (core) Alias to nvidia add-on
host-access # (core) Allow Pods connecting to Host services smoothly
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
rook-ceph # (core) Distributed Ceph storage using Rook
root@microk8s-01:~#
增加worker节点
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
# 在 microk8s-01 上执行
root@microk8s-01:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.248:25000/7e4a9f2bbd6bbc8242c101698b8b1296/a28774bf36f6
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.1.248:25000/7e4a9f2bbd6bbc8242c101698b8b1296/a28774bf36f6 --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.248:25000/7e4a9f2bbd6bbc8242c101698b8b1296/a28774bf36f6
microk8s join 172.17.0.1:25000/7e4a9f2bbd6bbc8242c101698b8b1296/a28774bf36f6
# 我做了一个新虚拟机 microk8s-05(ip:192.168.1.252)，执行，增加 --worker 参数
root@microk8s-05:~# microk8s join 192.168.1.248:25000/7e4a9f2bbd6bbc8242c101698b8b1296/a28774bf36f6 --worker
Contacting cluster at 192.168.1.248
The node has joined the cluster and will appear in the nodes list in a few seconds.
This worker node gets automatically configured with the API server endpoints.
If the API servers are behind a loadbalancer please set the '--refresh-interval' to '0s' in:
/var/snap/microk8s/current/args/apiserver-proxy
and replace the API server endpoints with the one provided by the loadbalancer in:
/var/snap/microk8s/current/args/traefik/provider.yaml
Successfully joined the cluster.
root@microk8s-05:~#
# 在 microk8s-05 上查看状态会提示要到控制节点去执行
root@microk8s-05:~# microk8s status
This MicroK8s deployment is acting as a node in a cluster.
Please use the control plane node.
# 在 microk8s-01 上查看状态，会发现并没有啥变化，主节点还是3台
root@microk8s-01:~# microk8s status
microk8s is running
high-availability: yes
datastore master nodes: 192.168.1.248:19001 192.168.1.249:19001 192.168.1.250:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
metrics-server # (core) K8s Metrics Server for API access to service metrics
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
gpu # (core) Alias to nvidia add-on
host-access # (core) Allow Pods connecting to Host services smoothly
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
rook-ceph # (core) Distributed Ceph storage using Rook
root@microk8s-01:~#
添加worker节点并不会增加master节点和standby节点。
添加更多节点
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# 在 microk8s-01 上执行
root@microk8s-01:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.248:25000/dd43e54eb0a87c8eed6e85b33fdd7d7d/a28774bf36f6
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.1.248:25000/dd43e54eb0a87c8eed6e85b33fdd7d7d/a28774bf36f6 --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.1.248:25000/dd43e54eb0a87c8eed6e85b33fdd7d7d/a28774bf36f6
microk8s join 172.17.0.1:25000/dd43e54eb0a87c8eed6e85b33fdd7d7d/a28774bf36f6
# 新创建的机器 microk8s-06（ip:192.168.1.253） 上执行，不带 --worker 参数
root@microk8s-06:~# microk8s join 192.168.1.248:25000/dd43e54eb0a87c8eed6e85b33fdd7d7d/a28774bf36f6
Contacting cluster at 192.168.1.248
Waiting for this node to finish joining the cluster. .. .. .. .
Successfully joined the cluster.
root@microk8s-06:~#
# 在 microk8s-01 上执行
root@microk8s-01:~# microk8s status
microk8s is running
high-availability: yes
datastore master nodes: 192.168.1.248:19001 192.168.1.249:19001 192.168.1.250:19001
datastore standby nodes: 192.168.1.253:19001
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
metrics-server # (core) K8s Metrics Server for API access to service metrics
registry # (core) Private image registry exposed on localhost:32000
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
cis-hardening # (core) Apply CIS K8s hardening
community # (core) The community addons repository
gpu # (core) Alias to nvidia add-on
host-access # (core) Allow Pods connecting to Host services smoothly
ingress # (core) Ingress controller for external access
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
nvidia # (core) NVIDIA hardware (GPU and network) support
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
rook-ceph # (core) Distributed Ceph storage using Rook
可以看到，新创建的机器不再是master节点，而是 standby 节点。