How certificates are used by k8s

2020-08-10 14:49:12

k8s需要多少证书呢，参考这里

主要包括

  • etcd 包括 etcd server、etcd peer、etcd client
  • kube-apiserver 需要kube-apiserver server证书
  • kube-scheduler 、kube-controller-manager、kube-proxy、kubelet　需要kube-apiserver client证书
  • kube-controller-manager 　需要service account证书
  • kubelet 需要kubelet server证书
  • kube-apiserver 需要kubelet client证书

由于验证证书只能指定一共root ca,需要同一系统内的证书都是同一个ca签署

TLS bootstrapping 简化kubelet证书制作

kubelet第一次启动，先使用bootstarp token,并将token设置属于system:bootstrappers,在认证通过后，kubelet申请
到kubelet server、kube-apiserver client for kubelet证书，之后在用证书做认证，取得kubelet权限，并且可以自动更新，具体可以参考这里

为了安全和审计，每个kubelet都有服务端和客户端两组证书，服务端和客户端与所在机器ip绑定，防止伪造认证证书
在kubelet启动后，本地的bootstrap token会被删除。

证书组件

  • etcd：使用 ca.pem、server-key.pem、server.pem
  • kube-apiserver：使用 ca.pem、server-key.pem、server.pem
  • kubelet：使用 ca.pem
  • kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem
  • kubectl：使用 ca.pem、admin-key.pem、admin.pem
  • kube-controller-manager：使用 ca-key.pem、ca.pem

cfssl是cloudflare开发的一个开源的PKI工具,k8s中证书只需要创建一次，以后在向集群中添加新节点时只要将/etc/kubernetes/ssl目录下的证书拷贝到新节点上即可

1
2
3
4
5
6
7
8
9
10
11
12
13
mkdir /data/ssl -p

curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o cfssl
chmod +x cfssl
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o cfssljson
chmod +x cfssljson
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl-certinfo_1.4.1_linux_amd64 -o cfssl-certinfo
chmod +x cfssl-certinfo

#移动到 /usr/local/bin目录下
mv cfssl /usr/local/bin/cfssl
mv cfssljson /usr/local/bin/cfssljson
mv cfssl-certinfo /usr/bin/cfssl-certinfo

先创建ca证书，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
[root@master01 ssl]# cat cert.sh 
#创建证书配置文件
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "expiry": "87600h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
       }
     }
   }
}
EOF
#创建CA证书签名请求文件
cat > ca-csr.json <<EOF
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
          "C": "CN",
          "L": "BeiJing",
          "ST": "BeiJing",
          "O": "k8s",
          "OU": "System"
        }
    ],
    "ca": {
        "expiry": "87600h"
    }
}
EOF
#生成CA证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------
#创建kubernetes证书签名请求文件 kubernetes-csr.json
cat > server-csr.json <<EOF
{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.1.101",
      "192.168.1.102",
      "192.168.1.103",
      "192.168.1.104",
      "192.168.1.105",
      "192.168.1.106",
      "192.168.1.31",
      "10.10.0.1",
      "lb.abc.com.cn",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "keys": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
     ]
}
EOF
#生成kubernetes证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

#-----------------------
#创建admin证书签名请求文件admin-csr.json
cat > admin-csr.json <<EOF
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
   },
   "names": [
     {
       "C": "CN",
       "L": "BeiJing",
       "ST": "BeiJing",
       "O": "system:masters",
       "OU": "System" 
     }
  ]
}
EOF
#生成admin证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#-----------------------
#创建 kube-proxy 证书签名请求文件 kube-proxy-csr.json
cat > kube-proxy-csr.json <<EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
   },
   "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
#生成kube-proxy证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy


cat <<EOF > /opt/kubernetes/ssl/front-proxy-ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}
EOF

cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca

cat front-proxy-client-csr.json
{
    "CN": "front-proxy-client",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

cfssl gencert   -ca=front-proxy-ca.pem  -ca-key=front-proxy-ca-key.pem -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare front-proxy-client

经过上述操作，我们会用到如下文件：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
[root@master01 ssl]# ls | grep pem
admin-key.pem
admin.pem
ca-key.pem
ca.pem
kube-proxy-key.pem
kube-proxy.pem
front-proxy-ca.pem
front-proxy-client-key.pem
front-proxy-client.pem
server-key.pem
server.pem

[root@master01 ssl]#  cfssl-certinfo -cert server.pem 
{
  "subject": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "issuer": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "serial_number": "19274655049576197307596686114763105929787949774",
  "sans": [
    "lb.abc.com.cn",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local",
    "127.0.0.1",
    "192.168.1.101",
    "192.168.1.102",
    "192.168.1.103",
    "192.168.1.104",
    "192.168.1.105",
    "192.168.1.106",
    "192.168.1.31",
    "10.10.0.1"
  ],
  "not_before": "2020-08-11T02:00:00Z",
  "not_after": "2030-08-09T02:00:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "6E:10:7:8A:76:12:FB:BA:1D:E7:FE:1:64:F3:91:27:84:11:B1:2F",
  "subject_key_id": "7C:9:71:B8:4A:44:67:B3:AF:2A:7E:AF:6E:85:25:CA:45:42:F1:4B",
  "pem": "-----BEGIN CERTIFICATE-----\nMIID7DCCAtSgAwIBAgIUA2BOME/jWBL4uij8QyiWoDZJQs4wDQYJKoZIhvcNAQEL\nBQAwZTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0Jl\naUppbmcxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwpr\ndWJlcm5ldGVzMB4XDTIwMDgxMTAyMDAwMFoXDTMwMDgwOTAyMDAwMFowZTELMAkG\nA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0JlaUppbmcxDDAK\nBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwprdWJlcm5ldGVz\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAER9TcihsMPkH269/9NFBZq0pghd4T\n2IdWRmmpyA8THsaguBq8sWjwOfmXbm8SYMJvFJmcsdFRM4gGpeeCrCEsQKOCAV0w\nggFZMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH\nAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUfAlxuEpEZ7OvKn6vboUlykVC8Usw\nHwYDVR0jBBgwFoAUbhAHinYS+7od5/4BZPORJ4QRsS8wgdkGA1UdEQSB0TCBzoIY\nbGIud2hpc3RsZS5ydWlqaWUuY29tLmNuggprdWJlcm5ldGVzghJrdWJlcm5ldGVz\nLmRlZmF1bHSCFmt1YmVybmV0ZXMuZGVmYXVsdC5zdmOCHmt1YmVybmV0ZXMuZGVm\nYXVsdC5zdmMuY2x1c3RlcoIka3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVy\nLmxvY2FshwR/AAABhwSsEc++hwSsEc+/hwSsEc/AhwSsEc+9hwSsEc+8hwSsEc+7\nhwSsEc8nhwQKCgABMA0GCSqGSIb3DQEBCwUAA4IBAQBJxUjmgxDdGl2dxnCJIS5B\nOSG2OqTtVi3TygkZESGKbD264TMD7VzrxXORSgxv87lEOrj6ktz4/gFLNJ19Z3gF\n3bl0DqCOO/ONHNnTLtxto/23dOF2KKMzERaJreFN1a1KyIBAGCyJdBQvQLwiYydj\nfnLxHmJKELUrCOZ8YLtYnHFm/24VcFoeqSHMCC+Csh0gsSHFTml9SJILQO7y+5uU\nD4ym2Hg5JPXmyW7CubsS6snTy9RgW0ptXoiCvIPdyIAtq7uEhHPXbNwAv21dXUg7\n/TLqimS6UdBVopsR5Fv9NKKvhoysHtQ9FPN/xwF/eezcXtJnKGONmi+NX8/Aq6ES\n-----END CERTIFICATE-----\n"
}

搭建k8s集群的时候，将这些文件分发到至此集群中其他节点机器中即可
kubeadm 证书期限有两种方案；第一种直接修改源码；第二种在启动集群时调整 kube-controller-manager 组件的 –experimental-cluster-signing-duration 参数，集群创建.可以查看参考链接

参考
kubeadm 证书期限调整
设置kubeadm自签证书为100年