Finisky Garden

I have a sharded cluster (2 shards, each 3 mongods; 3 config server, 2 mongoses) which is deployed by MongoDB Ops Manager.

Last week, one of the shard host status was shown as a grey diamond (Hover: "Last Ping: Never"). Besides, in the Ops Manager's server page, a server had two processes (e.g. sharddb-0 and sharddb-config). However, the cluster still works well and we can list the host sharddb-0-0(shard 0, replica 0) in the mongo shell by sh.status() and rs.status(). What's wrong with the cluster?

在一个分片MongoDB集群上并发执行transaction时遇到许多MongoCommandException错误: code 251, codename NoSuchTransaction:

Command find failed: cannot continue txnId 4 for session 38604515-2584-45a5-a17a-5eb5d34ea6c4 - = with txnId 5. Command find failed: cannot continue txnId 4 for session 38604515-2584-45a5-a17a-5eb5d34ea6c4 - = with txnId 6. Command insert failed: cannot continue txnId 31 for session 3ed7ea61-eae1-440f-8d95-b6e066b35b69 - = with txnId 34.

When I execute MongoDB transactions in parallel, I encounter lots of MongoCommandException: code 251, codename NoSuchTransaction:

Command find failed: cannot continue txnId 4 for session 38604515-2584-45a5-a17a-5eb5d34ea6c4 - = with txnId 5. Command find failed: cannot continue txnId 4 for session 38604515-2584-45a5-a17a-5eb5d34ea6c4 - = with txnId 6. Command insert failed: cannot continue txnId 31 for session 3ed7ea61-eae1-440f-8d95-b6e066b35b69 - = with txnId 34.

Lightsquid is a handy log analyzer for squid proxy. However, the project is not maintained since 2009.

Today, I found lightsquid doesn't work in 2021. Why?

MongoDB sharded cluster is the most complicated architecture. The deployment of sharded cluster in Kubernetes is relatively hard. We will go through the deployment process by MongoDB Ops Manager in this post.

Before start, please go through the Create a UserDB ReplicaSet first.

A MongoDB sharded cluster consists of the following components: - shard: Each shard contains a subset of the sharded data. Each shard can be deployed as a replica set. - mongos: The mongos acts as a query router, providing an interface between client applications and the sharded cluster. - config servers: Config servers store metadata and configuration settings for the cluster.

In this post, we are going to create a sharded cluster with 2 shards (3 instances replica set), 2 mongos and 3 config servers.

MongoDB Ops Manager Series:

1. Install MongoDB Ops Manager
2. Create a UserDB ReplicaSet
3. Expose UserDB to Public
4. Openssl Generates Self-signed Certificates
5. Enable UserDB TLS and Auth

Sharded cluster是MongoDB部署中最复杂的形式，因为Sharded cluster的组件较多，部署步骤也更为繁琐。在实际部署中还有几个部署ReplicaSet时没遇到的证书和TLS问题。阅读本文前，强烈建议先阅读 创建用户数据库(replicaset) 作为基础。

一个Sharded cluster由三部分构成：

• shard server: 存储一部分数据，每个shard可由一个replica set构成。
• mongos: query router，可以认为是整个cluster的前端，客户端通过mongos与cluster交互。
• config server: 存储cluster的metadata。

本文以创建一个2个Shard（每个由3实例ReplicaSet构成），2个mongos及3个config server的sharded cluster为例，演示具体部署流程。本文可做为Kubernetes部署MongoDB集群的番外篇。

整个系列：

1. 安装MongoDB Ops Manager
2. 创建用户数据库(replicaset)
3. 用户数据库服务配置公网访问
4. openssl生成自签名CA证书和server证书
5. 打开用户数据库TLS通信加密和Auth授权

This is part5, we will use the generated certficates to enable user database TLS and AUTH.

MongoDB Ops Manager Series:

1. Install MongoDB Ops Manager
2. Create a UserDB ReplicaSet
3. Expose UserDB to Public
4. Openssl Generates Self-signed Certificates
5. Enable UserDB TLS and Auth

This is part4, we will create a self-signed CA certificate and three server certificates.

MongoDB Ops Manager Series:

1. Install MongoDB Ops Manager
2. Create a UserDB ReplicaSet
3. Expose UserDB to Public
4. Openssl Generates Self-signed Certificates
5. Enable UserDB TLS and Auth

Self-signed certificates is not recommended for production. It cannot prevent man-in-the-middle attack. Since our main purpose is to encrypt the communication messages instead of authentication. Self-signed certificates is acceptable.

This is part3, we will expose the user database pods to the public so that Mongo client is able to access it.

MongoDB Ops Manager Series:

1. Install MongoDB Ops Manager
2. Create a UserDB ReplicaSet
3. Expose UserDB to Public
4. Openssl Generates Self-signed Certificates
5. Enable UserDB TLS and Auth

So far, the user database can be accessed only inside the kubernetes cluster. The official blog's approach is to expose the pods by NodePort: # Connect to a MongoDB Database Resource from Outside Kubernetes

I don't know why the official blog recommend NodePort. Obviously, LoadBalancer is a better way to expose MongoDB. We will use LoadBalancer to expose userdb pods.

This is part2, we will create a user database that is a 3 instances ReplicaSet.

MongoDB Ops Manager Series:

1. Install MongoDB Ops Manager
2. Create a UserDB ReplicaSet
3. Expose UserDB to Public
4. Openssl Generates Self-signed Certificates
5. Enable UserDB TLS and Auth

The so called Application Database is the backend DB of Ops Manager. It cannot be used to store user data. The user database is called MongoDB Deployment. Note that the deployment is different with Kubernetes deployment.

For simplicity, we will create a replicaset instead of a sharded cluster. The configuration process is almost the same. Because sharded cluster consists of more components, the configuration is more complicated. Let's start with an easier one. :-)

Reference: Deploy MongoDB Sharded Cluster by Ops Manager

Unfortunately, creating user database cannot be directly done by the Ops Manager web UI. Most of the operations should be performed by the MongoDB Kubernetes Operator, aka the kubectl command line. If you try to create it through web UI, you need to manually configure variety of parameters and install Agents which I don't know how to do it. So to my understanding, Ops Manager is a great tool for visualization and automation tasks, but not good at cluster manipulation. I think the reason is that the cluster specification is too complicated to incorperated to the UI.