安装 kubeadm：
sudo apt-get install -y kubelet kubeadm kubectl首先说一下镜像下载的问题，一开始找了几个，发现还是阿里的这个可用，别折腾了就用阿里的就行。用 registry.aliyuncs.com/google_containers 而不是 hangzhou 开头的那个，还需要注意的一点是 docker tag的时候，注意tag 是 registry.k8s.io/pause 开头，而不是 k8s.gcr.io/pause 可能和不同的版本有关，我用的是3.9的所以是 registry开头的。
主要取决于`kubeadm config images list` 返回：
~ kubeadm config images list
registry.k8s.io/kube-apiserver:v1.28.1
registry.k8s.io/kube-controller-manager:v1.28.1
registry.k8s.io/kube-scheduler:v1.28.1
registry.k8s.io/kube-proxy:v1.28.1
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1
部安装好后运行的容器有：
~ docker ps | grep ku
43f3dd223c6e kubernetesui/metrics-scraper "/metrics-sidecar" 4 hours ago Up 4 hours k8s_dashboard-metrics-scraper_dashboard-metrics-scraper-5657497c4c-kg5d5_kubernetes-dashboard_aa6a2bbb-4353-465e-8329-daca97fbbd1d_0
5eaec1f1885f kubernetesui/dashboard "/dashboard --insecu…" 4 hours ago Up 4 hours k8s_kubernetes-dashboard_kubernetes-dashboard-78f87ddfc-hjmdg_kubernetes-dashboard_d984a24e-6dd0-4fe1-be5e-b10c0f96cca3_0
7c8b2b0f1486 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 4 hours ago Up 4 hours k8s_POD_dashboard-metrics-scraper-5657497c4c-kg5d5_kubernetes-dashboard_aa6a2bbb-4353-465e-8329-daca97fbbd1d_0
8e9261aa7dc3 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 4 hours ago Up 4 hours k8s_POD_kubernetes-dashboard-78f87ddfc-hjmdg_kubernetes-dashboard_d984a24e-6dd0-4fe1-be5e-b10c0f96cca3_0
a84cdcd3060f ead0a4a53df8 "/coredns -conf /etc…" 5 hours ago Up 5 hours k8s_coredns_coredns-66f779496c-zcf56_kube-system_2e53844e-ab53-451c-97c6-3d39f01873c2_0
c7c5f219c2c8 ead0a4a53df8 "/coredns -conf /etc…" 5 hours ago Up 5 hours k8s_coredns_coredns-66f779496c-qz7s8_kube-system_20a35f8d-e026-4f04-a49c-0407b93845c5_0
6ca2c152ab91 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_coredns-66f779496c-qz7s8_kube-system_20a35f8d-e026-4f04-a49c-0407b93845c5_162
d9c7a137612d registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_coredns-66f779496c-zcf56_kube-system_2e53844e-ab53-451c-97c6-3d39f01873c2_140
7a02d759571f d73868a08083 "/opt/bin/flanneld -…" 5 hours ago Up 5 hours k8s_kube-flannel_kube-flannel-ds-hmg5z_kube-flannel_15864359-7159-41d0-a20b-ad61c5719f86_0
bec36f0de5f0 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_kube-flannel-ds-hmg5z_kube-flannel_15864359-7159-41d0-a20b-ad61c5719f86_0
3df3fd309adb 6cdbabde3874 "/usr/local/bin/kube…" 5 hours ago Up 5 hours k8s_kube-proxy_kube-proxy-pn5kl_kube-system_e041867c-a532-44ae-abbf-db23f85fdf61_0
ff25315a8c03 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_kube-proxy-pn5kl_kube-system_e041867c-a532-44ae-abbf-db23f85fdf61_0
152e6a83be5f 73deb9a3f702 "etcd --advertise-cl…" 5 hours ago Up 5 hours k8s_etcd_etcd-yangyan-linux_kube-system_1fefc5871a7eddfce29de48ad575e3dc_0
2a4b34745427 5c801295c21d "kube-apiserver --ad…" 5 hours ago Up 5 hours k8s_kube-apiserver_kube-apiserver-yangyan-linux_kube-system_182053f36b75c7913cb63fc07ca73fd3_0
66718debd9a7 b462ce0c8b1f "kube-scheduler --au…" 5 hours ago Up 5 hours k8s_kube-scheduler_kube-scheduler-yangyan-linux_kube-system_d5fdb9d1afd6c19d7d407600f2fa0639_0
3eb2ad8f8a81 821b3dfea27b "kube-controller-man…" 5 hours ago Up 5 hours k8s_kube-controller-manager_kube-controller-manager-yangyan-linux_kube-system_323d7d737f4dbe0c3fcf404ae75c1eed_0
babd2ebc9499 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_kube-scheduler-yangyan-linux_kube-system_d5fdb9d1afd6c19d7d407600f2fa0639_0
06e32960f366 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_kube-controller-manager-yangyan-linux_kube-system_323d7d737f4dbe0c3fcf404ae75c1eed_0
cb8fc520fc29 registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_etcd-yangyan-linux_kube-system_1fefc5871a7eddfce29de48ad575e3dc_0
41ba30006b2d registry.aliyuncs.com/google_containers/pause:3.9 "/pause" 5 hours ago Up 5 hours k8s_POD_kube-apiserver-yangyan-linux_kube-system_182053f36b75c7913cb63fc07ca73fd3如果打算用Docker作为容器，就使用cri-dockerd这个组建对接CRI。
git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
go build -o bin/cri-dockerd
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
sudo install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service如果不用Docker，用Containerd作为容器，不需要安装 cri-dockerd。下面的命令里也就不用指定 –cri-socket参数。
sudo kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock --kubernetes-version=v1.28.1在安装过程中，遇到了一些问题，修改了一些配置文件，贴出来，做个记录：
/etc/systemd/system/cri-docker.service 中修改：
ExecStart=/usr/local/bin/cri-dockerd –container-runtime-endpoint fd:// –network-plugin=cni –pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
下面是正常情况下产生的日志：
➜ ~
[init] Using Kubernetes version: v1.28.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local yangyan-linux] and IPs [10.96.0.1 192.168.1.6]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost yangyan-linux] and IPs [192.168.1.6 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost yangyan-linux] and IPs [192.168.1.6 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.002916 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node yangyan-linux as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node yangyan-linux as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: b2x0d4.FIREGOD.CN
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.1.6:6443 --token b2x0d4.FIREGOD.CN \
--discovery-token-ca-cert-hash sha256:79d54e0af3bf534a3ecce300dd949cbd462fcfiregode6d25e3d60bdf36e3e7f 启动后`kubectl get node`发现没有Ready，需要安装 flannel:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml因为为需要在tailscale网络中共享给我的笔记本使用，所以需要加入一个别名给ApiServer证书，原始文件可以使用导出命令（但是我注意到再次到处后并不是修改后的）：
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm-config.yml将修改后的配置保存到文件（我增加了mylinux）
apiServer:
certSANs:
- mylinux
- kubernetes
- kubernetes.default
- kubernetes.default.svc
- kubernetes.default.svc.cluster.local
- yangyan-linux
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.28.1
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/16
scheduler: {}执行重新生成证书：
sudo kubeadm init phase certs apiserver --config=/home/yangyan/kubeadm-config.yml在执行的时候，我遇到一个问题，总是提示有多个cri-socket，让我指定第一个，但是上面这个重新生成证书的指令并不支持 `–cri-socket`
unknown flag: --cri-socket
To see the stack trace of this error execute with --v=5 or higher实在是没有办法，将containerd卸载后，才运行过去（证书生成之后为又装回来了），这里有一个前提是为想要用docker的容器，所以才卸载了containerd，但是在为生成证书成功后。
我无意中发现一个问题，发现原来在我docker中的jenkins挂了之后无法启动了，看docker日志，发现docker 它用到 containerd.sock
Sep 03 18:19:28 yangyan-linux dockerd[7754]: time="2023-09-03T18:19:28.258324467+08:00" level=error msg="collecting stats for a84cdcd3060fdecfef9718e055f33d64085d62d066b65d2055c74e3099678a6b: connection error: desc = \"transport: Error while dialing dial unix:///run/containerd/containerd.sock: timeout\": unavailable"所以后来为又把containerd安装后启用了。
检查一下证书，证明已经有了：
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep DNS
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:mylinux, DNS:yangyan-linux, IP Address:10.96.0.1, IP Address:192.168.1.6