About safety

Enable authentication for the MongoDB cluster

Authentication is not enabled for the MongoDB server by default. This means that anyone can connect directly to the Mongod instance and perform arbitrary database operations.

Assign different role rights to different users

MongoDB supports role-defined permission systems. You should explicitly assign users only the required permissions based on the “minimum permissions” rule.

Use a central authentication server

Use a central authentication server such as LDAP and Kerbero whenever possible, and use a strong password policy.

Create a whitelist for application servers that need access to MongoDB (firewall configuration)

If your server has multiple network adapters, you are advised to listen for services only on Intranet IP addresses.

Use an encryption engine for sensitive data

MongoDB Enterprise edition supports storage encryption, and encryption engines should be used to protect sensitive data involving customers.

About the deployment

Use a replicate set of at least three data nodes

The recommended minimum deployment of MongoDB is a replication set consisting of three data nodes. Replication sets can provide the following benefits:

  • System 99.999% high availability
  • Automatic failover
  • Data redundancy
  • Disaster deployment
  • Reading and writing separation

Don’t shard too early

Sharding can be used to extend the read and write capabilities of your system, but sharding also introduces new challenges such as administrative complexity, increased cost, and the challenge of selecting the right shard keys. Generally speaking, you should consider sharding after exhausting other performance tuning options, such as index optimization, pattern optimization, code optimization, hardware resource optimization, IO optimization, etc.

Select the appropriate number of shards

Some trigger conditions for sharding are:

  • The amount of data is too large to manage on one server
  • The concurrency is too high for one server to handle in a timely manner
  • The disk I/O pressure is too high
  • The memory of a single system is not large enough to store hot data
  • The server NIC processing capacity reaches the bottleneck. Procedure
  • Depending on your shard trigger conditions, you can determine the number of shards required by taking the total requirement and dividing it by the capacity of each server.

Deploy enough replica set members for each shard

Data between shards does not copy each other. Data for each shard must be highly available within the shard. Therefore, at least three data nodes must be deployed for each shard to ensure that the shard will not be unavailable due to the breakdown of the primary node.

Select the appropriate slice key

In the sharding scenario, one of the most important considerations is to select the appropriate shard key. The choice of slice keys takes into account the read/write mode of the application. Typically a slice key is optimized for either write or read operations. There are trade-offs based on which operations are more frequent.

  • The slice key value should have a high cardinality, or that the slice key has many different values in the set. For example, _id is a slice key with a high cardinality because the _id value does not repeat
  • A slice key should not grow continuously, as timestamp is a slice key that grows continuously. This kind of slice key tends to cause hot sharding, that is, new writes are concentrated in a shard
  • A good shard key should direct the query to one (or more) shards to improve query efficiency. In general, this means that the slice key should include the fields used by the most common queries
  • Good slice keys should be spread out enough so that new inserts can be spread across multiple shards to increase concurrent write rates.
  • You can use combinations of several fields to form slice keys for several different purposes (cardinality, dispersion, query orientation, etc.)

About a system

Use SSDS or RAID10 to improve storage IOPS

MongoDB is a high performance and high concurrency database, most of its I/O operations are random updates. Generally speaking, native SSDS are the best storage solution. If common hard disks are used, RAID10 striping is recommended to improve the concurrency of I/o channels.

Use separate physical volumes for Data and Journal/log

Many of MongoDB’s performance bottlenecks are IO related. You are advised to create an independent physical volume for the log disks (Journal and system logs) to reduce the I/O resource occupation on the data disks. System logs can be specified directly on the command line or in configuration file parameters. Journal logs cannot be assigned to another directory directly. This can be resolved by creating a Symbol link to the Journal directory.

Use the XFS file system

MongoDB recommends using XFS file system in WiredTiger storage engine. Ext4 is the most common, but does not perform well under IO pressure due to internal journal conflicts with WiredTiger.

Use large caches with caution under WiredTiger

WiredTiger drops write operations asynchronously. The default checkpoint is 60 seconds. To checkpoint, you need to traverse all the dirty data in memory to clean up and write the data to hard disk. If the cache is large (for example, larger than 128 GB), the checkpoint time takes a long time. Data writing performance is affected during checkpoint. Currently, you are advised to set the actual cache size to 64GB or less.

Disable Transparent Huge Pages

Transparent Huge Pages (THP) is a Linux memory management optimization that reduces Translation Lookaside Buffer(TLB) overhead by using larger memory Pages. Most of the MongoDB database is relatively scattered and small amount of data read and write, THP will have a negative impact on the MongoDB working condition, so it is recommended to close. Docs.mongoing.com/manual-zh/t…

To enable the Log Rotation

This prevents the MongoDB log file from expanding indefinitely and occupying too much disk space. A good practice is to enable the log rotation and clean up the history log files as soon as possible.

Allocate enough Oplog space

Sufficient Oplog space ensures that you have enough time to restore a slave node from scratch or perform time-consuming maintenance operations on the slave node. Assuming your longest offline maintenance operation takes H hours, your Oplog should be guaranteed to hold at least H 2 or H3 hours of Oplog.

Disable atime for database files

Preventing the system from updating the access time of files can effectively improve the performance of file reading. This can be done by adding the noatime parameter to the /etc/fstab file. For example, run the /dev/xvdb/data ext4 noatime 0 0 command to mount the file again, run the mount -o remount /data command

Increase default file descriptors and process/thread limits

Linux’s default number of file descriptors and maximum number of processes is generally too low for MongoDB. It is recommended to set this value to 64,000. Because the MongoDB server needs a file descriptor for each database file and each client connection. If this number is too small it may cause errors or unresponsiveness in large concurrent operations. You can modify these values by using the following command: ulimit -n 64000 ulimit -u 64000

Prohibit the NUMA

On a multiprocessor Linux system that uses NUMA technology, you should disable NUMA. MongoDB performance can sometimes be slow in NUMA environments, especially when process loads are high.

Readahead setting

The prefetch value is an optimization of the file operating system, which basically means that when a program requests a page, the file system reads several pages at the same time and returns them. The reason for this is that many times IO is the most time-consuming disk seek. With prefetch, the system can return the following data in advance at the same time. This saves a lot of disk seek time, assuming the program is doing a continuous read. A lot of times MongoDB does random access. For random access, this prefetch value should be set to a small value. Generally 32 is a good choice. You can use the following command to display the prefetch value of the current system: blockdev –report To change the prefetch value, use the following command: blockdev –setra 32

Use the NTP time server

When using the MongoDB replication set or sharded cluster, ensure that the NTP server is used. This ensures proper synchronization between MongoDB cluster principles.

About the index

Create appropriate indexes for each of your queries

This is for large data volumes such as more than tens of millions (number of documents) of the order of magnitude. Without indexing MongoDB needs to read all documents from disk into memory, which can put a lot of strain on the MongoDB server and affect the execution of other requests.

Create appropriate composite indexes and do not rely on cross-indexes

If your query will use multiple fields, MongoDB has two indexing techniques to use: cross-indexing and composite indexing. Cross-index is to set up a single field index for each field, and then use the corresponding single field index to cross the query results during query execution. Cross indexing has a low trigger rate, so if you have a multi-field query, it is recommended to use composite indexes to ensure the normal use of the index. For example, if the application needs to find all athletes younger than 30: db.athelets. Find ({sport: “marathon”, location: “sz”, age: {$lt: 30}}}) then you might need an index like db.athelets. EnsureIndex ({sport:1, location:1, age:1});

Combination index field order: Equality First, Range After

As an example, if the condition is a match or a range when creating a composite index, then the match condition (sport: “marathon”) should precede the composite index. The range condition (AGE: <30) field should be placed after the composite index.

Use Covered indexes whenever possible

Sometimes your query returns only a few or even a single field, for example, if you want to find the destinations of all flights departing from Hongqiao Airport. Db.flights.find ({origin:”hongqiao”}, {dest:1});} db.flights.find({origin:”hongqiao”}, {dest:1}); Such queries contain the _ID field by default, so you need to scan the matching documents and retrieve the results. Instead, use the query: db.flights.find({origin:”hongqiao”}, {_id:0, dest:1}); MongoDB can retrieve all values that need to be returned directly from the index without having to scan the actual document (which may need to be fetched from disk into memory).

Indexing is run in the background

When a collection is indexed, the database on which the collection resides will not accept other reads and writes. To index a collection of data, use the background option {background: true}

Application configuration

Set appropriate MongoDB pool size (Connections Per Host)

The default connection pool size for Java drivers is 100. You are advised to adjust the value based on the actual application situation. For applications with low pressure, you can adjust the value to reduce the resource occupation on the application server.

Use Write Concern correctly

The recommended minimum deployment for MongoDB is a replicate set with three data nodes. Write operations (update, insert, or delete) that are applied by default are returned immediately after completion on the primary node. Write operations are asynchronously copied to other nodes in the background through OPLOG. In extreme cases, the master node may go down before these writes are copied to the slave node. At this time, an active/standby node switchover occurs. The write operations on the original active node are rolled back to the file and are invisible to the application. To prevent this, MongoDB recommends using the {w: “marjority”} option for important data. {w: “majority”} ensures that data is copied to most nodes before a successful result is returned. Using this mechanism can effectively prevent data rollback. Alternatively, you can use {j:1} (this can be combined with w: “majrotiy”) to specify that data must be written to WAL before a success acknowledgement is returned to the application. This can cause write performance to degrade, but can be considered for important data.

Use Read Preference Settings correctly

Because MongoDB is a distributed system, a piece of data is replicated on multiple nodes. The node from which data is read depends on the data read requirements of the application. The following are the read options that can be configured in a set:

  • Primary: Data is read on the primary node by default
  • PriaryPreferred: Read data from the primary node first. If the data is read successfully, read data from any secondary node
  • Secondary: read data from the secondary node (if there are multiple nodes, use one of the secondary nodes randomly)
  • SecondaryPreferred: Read data from the secondary node first. If the secondary node cannot provide services for some reason, read data from the primary node
  • Nearest: read from the nearest node. The distance is determined by the time of the ping operation. Except for the first option, there is a possibility that the read data is not up to date. The reason is that data replication is done asynchronously in the background.

Don’t instantiate multiple MongoClients

MongoClient is a thread-safe class with its own thread pool. Generally, don’t instantiate multiple MongoClient instances within a SINGLE JVM to avoid excessive connections and unnecessary waste of resources.

Use a Retry mechanism for write operations

MongoDB can achieve 99.999% high availability using replication set technology. If a primary node fails to write data, the system automatically fails over to another node. The migration may take a few seconds, during which the application should catch the corresponding Exception and retry. Retries should have a backoff mechanism, for example, retries at 1s, 2s, 4s, 8s, etc.

Avoid using long field names

MongoDB has no table structure definition. The structure of each document is determined by the fields inside each document. All field names are repeated within each document. Using field names that are too long can result in increased memory and network bandwidth requirements. (Due to compression technology, long field names do not occupy much storage on the hard disk)

Use projection to reduce the amount of content returned

MongoDB supports filtering of returned fields, similar to select in SQL statements. Using Projection can reduce the amount of content returned, the amount of network transmission, and the time it takes for code to be transformed into objects.

Use TTL to automatically delete expired data

Most of the time we use MongoDB to store some time-sensitive data, such as 7-day monitoring data. EnsureIndex ({create_time:1}, {expireAfterSeconds: 7*24*3600})