Kubernetes部署MongoDB集群(五):打开用户数据库TLS通信加密和Auth授权

本文为系列第五部分,用生成的自签名证书打开userdb的TLS和AUTH,并且完成userdb的公网域名访问。

整个系列:

  1. 安装MongoDB Ops Manager
  2. 创建用户数据库(replicaset)
  3. 用户数据库服务配置公网访问
  4. openssl生成自签名CA证书和server证书
  5. 打开用户数据库TLS通信加密和Auth授权

理解不同层面的TLS加密

开始之前,先解释下部署中不同层面的TLS加密:Secure Connections to Ops Manager, Secure Connections to Application Database, Secure Connections to MongoDB Deployments,我们需要的是最后一个,Enable TLS to MongoDB Deployments。而且这个配置不能直接通过界面完成,因为需要将生成的自签名证书“上传”到userdb的每个server上,然后才能配置这些证书所在的路径。

用一张图解释: Different Level of Security

如果你读Ops Manager的文档 # Enable TLS for a Deployment:

Get and Install the TLS Certificate on Each MongoDB Host

Acquire a TLS certificate for each host serving a MongoDB process. This certificate must include the FQDN for the hostname of this MongoDB host. The FQDN can be the Common Name or the Subject Alternative Name of this host. You must install this TLS certificate on the MongoDB host.

读完一头雾水,到底怎么安装这些证书到pod上呢?实际上,应该通过MongoDB Kubernetes Operator的 # Secure Deployments using TLS 打开TLS。

一个小插曲,开始做到这一步时,考虑直接把证书通过kubectl cp拷贝到statefulset pod上去。先用kubectl exec 进到pod中查看:

groups: cannot find name for group ID 2000
I have no name!@mongo-1:/$ cat > ca.crt
bash: ca.crt: Permission denied
I have no name!@mongo-1:/$ whoami
whoami: cannot find name for user ID 2000
I have no name!@mongo-1:/$ 
发现这个pod非常奇怪,没有用户,user和group都是2000这么个诡异的东西。再后来考虑过通过Mount Volume的方式将其加入到pod上,都不是正确的打开方式。

另一篇文章供参考: # Secure MongoDB Enterprise on Red Hat OpenShift

上传证书到Kubernetes集群

创建Secret存储Server证书

将第四部分生成的几个证书另存为:

Save As
rootca.crt ca-pem
userdbX.pem userdb-X-pem

其中X取值0,1,2,加上rootca共四个文件。注意此处重命名后的文件没有后缀。三个userdb的pem文件为如下格式:

-----BEGIN CERTIFICATE-----
...
... your TLS certificate
...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
...
... your private key
...
-----END RSA PRIVATE KEY----
而rootca.crt中只有证书段,没有私钥:
-----BEGIN CERTIFICATE-----
...
... your TLS certificate
...
-----END CERTIFICATE-----

创建存储三个server证书的Secret:

$ kubectl create secret generic userdb-cert --from-file=userdb-0-pem --from-file=userdb-1-pem  --from-file=userdb-2-pem -n mongodb
secret/userdb-cert created

创建ConfigMap存储CA证书

$ kubectl create configmap custom-ca --from-file=ca-pem -n mongodb
configmap/custom-ca created

更新cluster的replicaSetHorizons

回顾第三部分用户数据库服务配置公网访问的最后:

需要在userdb.yaml中使用spec.connectivity.replicaSetHorizons 指定公网访问的地址。而使用此选项又必须打开spec.security.tls

security:
    tls:
      enabled: true
  connectivity:
    replicaSetHorizons:
      - "userdb": "userdb0.com:27017"
      - "userdb": "userdb1.com:27017"
      - "userdb": "userdb2.com:27017"

终于来到打开TLS的最后一步,回到userdb.yaml,加入上面security的部分:

apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: userdb
  namespace: mongodb
spec:
  members: 3
  version: 4.2.2-ent
  type: ReplicaSet

  opsManager:
    configMapRef:
      name: ops-manager-connection
  credentials: ops-manager-admin-key
  security:
    tls:
      enabled: true
      ca: custom-ca
  connectivity:
    replicaSetHorizons:
      - "userdb": "userdb0.com:27017"
      - "userdb": "userdb1.com:27017"
      - "userdb": "userdb2.com:27017"
注意: replicaSetHorizons下面每行中的userdb,即每行的key并不重要,只要保证三行的key string是相同的即可。

应用userdb.yaml:

$ kubectl apply -f userdb.yaml -n mongodb
mongodb.mongodb.com/userdb configured

Updating UserDB Deployment

具体更新的Details: Updating Deployment Details 可见为了保证服务不中断,TLS的更新需要经过多步完成:disabled -> sslAllowed -> sslPreferred -> sslRequired

这步更新需要一些时间,要重启多次,中间DB对应的statefulset userdb还会重启。如果出现service匹配不到pod的情况,参考第三部分提到暴露service之后要把controller-revision-hash从service的spec.selector中删除。

完毕之后,Processes的界面TLS会变为Enabled。如果kubectl exec查看userdb对应的pod,可以发现证书保存在/mongodb-automation/

I have no name!@userdb-0:/$ ls -l /mongodb-automation/
total 8
lrwxrwxrwx 1 2000 root   45 Dec 11 06:25 ca.pem -> /var/lib/mongodb-automation/secrets/ca/ca-pem
drwxr-xr-x 2 2000 root 4096 Dec 11 06:25 files
-rw-r--r-- 1 2000 root    3 Dec 11 06:25 mongodb-mms-automation-agent.pid
lrwxrwxrwx 1 2000 root   54 Dec 11 06:25 server.pem -> /var/lib/mongodb-automation/secrets/certs/userdb-0-pem

Error calling ComputeState, Error dialing to connParams, Error checking if rs member is up错误

如果这步一直更新不成功,看Ops Manager的Agent Log出现如下错误,大概率是证书有问题,虽然看起来跟证书没什么关系:

<userdb-1> [03:29:59.660] Failed to compute states : <userdb-1> [03:29:59.660] Error calling ComputeState : <userdb-1> [03:29:59.660] Error getting fickle state for current state : <userdb-1> [03:29:59.660] Error checking if rs member = userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 is up : <userdb-1> [03:29:59.660] Error executing WithClientFor() for cp=userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false) connec tMode=SingleConnect : <userdb-1> [03:29:59.660] Error checking out client (0x0) for connParam=userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : [03:29:59.659] Error dialing to connParams=userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false): tried 4 identities, but none of them worked. They were (__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], __system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], mms-automation@admin[[MONGODB-CR/SCRAM-SHA-1]][24], )

Error checking out client (0x0) for connParam=userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false) connectMode=SingleConnect : [03:32:57.470] Error dialing to connParams=userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false): tried 4 identities, but none of them worked. They were (__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], __system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], mms-automation@admin[[MONGODB-CR/SCRAM-SHA-1]][24], )

Error getting client ready for conn params = userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false). Informing all requests and disposing of client (0x0). requests=[ 0xc001e58780 ] : [03:32:57.470] Error dialing to connParams=userdb-0.userdb-svc.mongodb.svc.cluster.local:27017 (local=false): tried 4 identities, but none of them worked. They were (__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], __system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][1024], mms-automation@admin[[MONGODB-CR/SCRAM-SHA-1]][24], )

大概率是因为在生成证书的时候,DNS里没有加入私网域名:userdb-0.userdb-svc.mongodb.svc.cluster.local。此时只能回到第四部分openssl生成自签名CA证书和server证书,修改userdb<X>.cnf中的[ alt_names ]段,加入私网域名,重新生成server证书,再更新userdb。

Mongo Client连接UserDB

成功的话,现在再用mongo shell连接数据库,注意要打开TLS并指定--tlsCAFile。这里也有一点小技巧,mongodb的connection string不支持指定tlsCAFile,所以要采用命令行和connection string相结合的用法:

$ mongo mongodb://userdb0.com:27017,userdb1.com:27017,userdb2.com:27017/ --tls --tlsCAFile rootca.crt
MongoDB shell version v4.4.2
connecting to: mongodb://userdb0.com:27017,userdb1.com:27017,userdb2.com:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("fcbaa78a-21dd-472f-a84f-bf3435bf9088") }
MongoDB server version: 4.2.2
WARNING: shell and server versions do not match
---
The server generated these startup warnings when booting:
2020-12-11T06:25:58.940+0000 I  STORAGE  [initandlisten]
2020-12-11T06:25:58.940+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-12-11T06:25:58.940+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-12-11T06:26:00.201+0000 I  CONTROL  [initandlisten]
2020-12-11T06:26:00.201+0000 I  CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-12-11T06:26:00.201+0000 I  CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2020-12-11T06:26:00.201+0000 I  CONTROL  [initandlisten]
---
MongoDB Enterprise userdb:PRIMARY>

最后的PRIMARY说明cluster配置成功!用rs.conf()确认一下horizons:

MongoDB Enterprise userdb:PRIMARY> rs.conf()
{
        "_id" : "userdb",
        "version" : 2,
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : true,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "userdb-0.userdb-svc.mongodb.svc.cluster.local:27017",
...
                        "horizons" : {
                                "userdb" : "userdb0.com:27017"
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "userdb-1.userdb-svc.mongodb.svc.cluster.local:27017",
...
                        "horizons" : {
                                "userdb" : "userdb1.com:27017"
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 2,
                        "host" : "userdb-2.userdb-svc.mongodb.svc.cluster.local:27017",
...
                        "horizons" : {
                                "userdb" : "userdb2.com:27017"
                        },
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
                }
        ],
...
}

打开AUTH

TLS设置成功之后打开AUTH就非常简单了,直接在Deployment的Processes界面中点AUTH的Disabled,按提示直接点下一步到最后Save,再点下界面上方的review & deploy即可。

一切完成之后的界面如下,可见TLS和AUTH都已变成了EnabledUserDB Deployment Completed

此时再用mongo shell登录不再有Startup Warnings:

$ mongo mongodb://userdb0.com:27017,userdb1.com:27017,userdb2.com:27017/ --tls --tlsCAFile rootca.crt
MongoDB shell version v4.4.2
connecting to: mongodb://userdb0.com:27017,userdb1.com:27017,userdb2.com:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("5063023a-b76d-4d03-923f-18332e2e7cc9") }
MongoDB server version: 4.2.2
WARNING: shell and server versions do not match
MongoDB Enterprise userdb:PRIMARY>

总结

本系列文章描述了如何部署MongoDB Ops Manager,并使用Ops Manager搭建用户MongoDB数据库的详细流程。中间步骤繁多,细节之处难免有疏漏,可在评论中留言指出。

感谢阅读!