MongoDB is getting shot a lot on Hack News recently. Many people claim to hate MongoDB, with David Mytton revealing many of MongoDB’s problems on his blog. Haters, however, have their own favourites, too, to counter: Russell Smith offers a summary of years of work experience. Russell Smith has worked as a scaling consultant for Ops and large websites and has helped Guardian, Experian, and others, and is co-founder of the MongoDB London User Group. As the MongoDB Master (a core contributor organization officially recognized by MongoDB and sharing its own professional technology through the community), the infrastructure that he participates in the work queries more than 30,000 times per second, and the active data is more than 1TB per day.
Here’s Russell’s analysis of some of MongoDB’s most common and unusual problems:
32 bits vs 64 bits
Most servers now support 32-bit operating systems, and newer hardware supports 64-bit operating systems that allow more RAM.
MongoDB also releases both 32-bit and 64-bit versions of the database. Due to the memory-mapped files used by MongoDB, the 32-bit version only supports 2 gigabytes of data. For the standard Replica Set, MongoDB only has a single processing strategy — Mongod. If you want to store more than 2 gigabytes of data in the future, use a 64-bit version of MongoDB. If you have a shard installation, the 32-bit version is also available.
Summary: Use the 64-bit version or understand the limitations of the 32-bit version.
File size limit
Unlike AN RDBMS, which stores data in rows and columns, MongoDB stores data in files. These files are stored in binary format in a JSON-like BSON format.
As with other databases, there is a limit to the size of a single file. In older versions of MongoDB, individual files were limited to 4M. New versions of MongoDB support single files up to 16 MB in size. This restriction can be annoying, but 10Gen has an opinion: If this setting keeps bothering you, is there a problem with your design pattern? Or you can use GridFS with unlimited file size.
The common advice in this case is to avoid storing large files and to update the various objects stored in the database irregularly. A service like Amazon S3 or Rackspace Cloudfiles is often a better choice than not overloading the infrastructure if it’s not necessary.
Bottom line: Keep each file under 16MB, and all is well.
Write failed
MongoDB allows fast writes and updates by default, at the cost of no explicit error notification. By default, most drivers do asynchronous, “unsafe” writes — which means that the driver cannot immediately respond with error information, similar to MySQL’s INSERT DELAYED. If you want to know if something is successful, you must manually check for error messages using getLastError.
In some cases if you need to get the error message immediately after the error occurs, it is easy to implement synchronous “safe” queries in most drivers. This kills the advantage that MongoDB has over traditional databases.
If you need a bit more performance than “completely safe” synchronous writes, and still want a degree of security, you can use getLastError with ‘j’ to make MongoDB only issue error notifications after a log commit. Instead of 60 seconds, logs will be output to disk once in 100 milliseconds.
Summary: If you must write validation, you can use safe write or getLastError.
The weakening of the data structure model does not mean the absence of the data structure model
RDBMSS typically have a predefined data structure model: rows and columns of a table, with each field having a name and data type. If you want to add a column to one row, you must add a column to the entire table.
MongoDB removes this setting without enforcing model constraints on collections and files. This is good for rapid development and easy modification.
This does not mean that you can ignore the structural model design, a proper structural model can give you the best performance of MongoDB. Read the MongoDB documentation, or watch videos on how to design these architectural models.
Conclusion: Design the structural model and take full advantage of MongoDB features.
By default, the modify statement modifies only a single file
In a traditional RDBMS, unless the LIMIT clause is used, the modification statement will apply to all matches. However, MongoDB uses the equivalent “LIMIT 1” setting by default on each query. You can’t do “LIMIT 5”, but you can remove the LIMIT entirely by saying:
db.people.update({age: {$gt: 30}}, {$set: {past_it: true}}, false, true)
There is also a similar option in the official driver – ‘Multi’.
Summary: Multi-file modification can be accomplished by specifying multi to true for multiple files
Query case sensitive
String queries may not evolve as expected — this is due to MongoDB being case-sensitive by default.
Db.people. find ({name: ‘Russell’}) is different from db.people.find ({name: ‘Russell’}). The ideal solution here is to validate the data to be queried. You can also query with regular expressions, such as db.people.find ({name:/Russell/ I}), but this affects performance.
Bottom line: Queries are case sensitive, and regular expressions can be used at the expense of speed.
No fault tolerance for input data
When you try to insert the wrong type of data into a traditional database, the traditional database will usually convert the data to a predefined type. However, this is not possible in MongoDB because MongoDB files do not have a predefined data model. That way MongoDB will insert whatever data you enter.
Summary: Use accurate data types.
About the lock
When a resource is shared by multiple parts of the code, make sure the lock ensures that the resource can only be operated on in one place.
Older versions of MongoDB (Pre 2.0) had a global write lock. This means that there is only one place to write across the entire server. This can cause the database to stall because of a lock overload somewhere. This issue has been significantly improved in version 2.0 and has been further enhanced in the current 2.2 release. MongoDB 2.2 takes this issue a big step further by using database-level locks. Collection level locks are also expected in the next release.
Nonetheless, Russell believes that most applications that are subject to this restriction are more directly influenced by MongoDB than by the application itself.
Bottom line: Use the latest stable release for maximum performance.
About the package
When installing on Ubuntu – and Debian – like systems, many people have experienced problems with “dated versions”. The solution is simple: using the official 10Gen library, installing on Ubuntu and Debian will be as smooth as installing on Fedora and Centos.
Summary: Use official packages that have most of the latest versions.
Use an even number of Replica Set members
Replica Set is an effective way to increase redundancy and improve the performance of MongoDB data cluster. The data is replicated among all nodes and one is selected as the master node. If the primary node fails, one of the other nodes is voted as the new primary node.
It is tempting to use two machines in the same Replica Set. It is cheaper than three machines and is standard RDBMS style.
But in MongoDB, the number of members in the same Replica Set can only be an odd number. If you use an even number of members, when the primary node fails all other nodes become read-only. This happens because the number of nodes left to choose does not meet the primary node voting requirement.
If you want to save money and still want to support failover and redundancy enhancement, you can use Arbiter. Arbiter is a special Replica Set member that does not store any user data (which means they can use very small servers).
Summary: Only an even number of Replica Set members can be used, but Arbitter can be used to cut costs.
No JOIN statement
MongoDB does not support joins: if you want to retrieve data in multiple collections, you have to do multiple queries.
If you feel that you are doing too many queries manually, you can redesign your data model to reduce the overall number of queries. Files in MongoDB can be of any type, so it’s easy to de-normalize data. This will keep it consistent with your application.
Summary: Take a look at how to design a data structure model without joins.
Journaling
MongoDB uses memory-mapped files and sends notifications to disk every 60 seconds, which means that at most you can lose all data for 60 seconds plus notifications to disk.
To avoid data loss, MongoDB has added Journaling since version 2.0 (on by default). The Journaling changed the time from 60 seconds to 100ms. If the database is unexpectedly down, it will be restarted prior to startup to ensure that the database is in a consistent state. This is where MongoDB comes closest to a traditional database.
Of course Journaling affects performance slightly, by about 5%. But for most people the extra security is certainly worth the price.
Bottom line: Better not shut down Journaling.
By default, there is no authentication
MongoDB does not have authentication by default. MongoDB thinks it’s in a trusted network with a firewall. But that doesn’t mean it doesn’t support authentication, which can be easily turned on if needed.
Conclusion: MongoDB security can be ensured by using firewalls and binding the right interfaces, as well as enabling authentication.
Data lost in the Replica Set
Using Replica Set is an effective way to improve system reliability and easy maintenance. In this case, it is very important to understand the occurrence and transition mechanism of faults between nodes.
Members in the Replica Set transmit information through oplog (which records the list of operations such as additions, deletions, and alterations in the data). When one member changes the Oplog, the other members follow the oplog. If the node you are responsible for processing the new data resumes after an error, it will be rolled back to the last oplog common point. However, in the process: The lost “new data” has been removed from the database by MongoDB and stored in your data directory “rollback” to be manually restored. If you don’t know this feature, you might think the data has been lost. This directory must be checked every time a member recovers from an error. Restoring this data is easy with standard tools released by MongoDB. Check the official documentation for more information.
Summary: Data lost during failover will appear in the ROLLBACK directory.
Shard too late
Sharding is to split data among multiple machines. It is usually used to improve performance when the Replica Set runs too slowly. MongoDB supports automatic sharding. However, if you leave sharding too late, problems arise. Because it takes time and resources to split the data and move the chunks, if the server is basically running out of resources it is likely that you will not be able to shard the data when you need it most.
The solution is simple: use a tool to monitor MongoDB. Make the most accurate assessment of your server and shard it before 80% of its overall performance. Similar monitoring tools are: MMS, Munin (+Mongo Plugin) and CloudWatch.
If you are sure you want sharding from the start, a better recommendation would be to use AWS or a similar cloud service for sharding. On a small server, shutting down or adjusting the machine is much more straightforward than moving thousands of blocks of data.
Conclusion: Sharding early is the best way to avoid problems.
You cannot change the Shard key in a file
For sharding Settings, the shard key is used by MongoDB to identify the corresponding files for sharding. When you insert a file, you cannot change the shard key of the file. The solution here is to delete the document and then rebuild it, allowing it to be assigned to the corresponding partition.
Summary: The shard key cannot be modified. If necessary, the shard key can be deleted and re-created.
A Collection larger than 256 GB cannot be sharded
Going back to the sharding too late issue — MongoDB does not allow sharding of collections that grow larger than 256GB, which was not available in previous versions. This restriction will definitely be removed in the future, and there is no better solution. You can only recompile or keep the size below 256GB.
Summary: Shard the Collection before it reaches 256GB.
Unique index and share
An index’s uniqueness constraint can only be guaranteed by the Shard key.
For more details
Incorrect Shard key was selected
MongDB requires you to select a shard key to fragment data. If you choose the wrong shard key, changing it can be cumbersome.
Click to see how to change
Summary: Read this document before selecting the Shard key.
Unencrypted communication with MongoDB
Connections to MongoDB are unencrypted by default, which means your data could be logged and used by third parties. This is unlikely to happen if your MongoDB is being used on your own non-WAN.
However, if you’re accessing MongoDB over a public network, you’ll want your communications to be encrypted. Public MongoDB does not support SSL. Fortunately, it is very easy to customize your own version. Users of 10Gen have a specially customized encrypted version. Fortunately, most official drivers support SSL, but minor hassles are inevitable. Click to view documentation.
Conclusion: When connecting to a public network, note that communication with MongoDB is not encrypted.
Unlike traditional databases such as MySQL, which support atomic operations on multiple rows of data, MongoDB only supports atomic changes to a single file. One way to solve this problem is to use asynchronous commits in your application; Another is to create more than one data store. Although the first method is not suitable for every situation, it is obviously better than the second method.
Summary: Transactions to multiple files are not supported.
Log pre-allocation is slow
MongDB might tell you it’s ready, but it’s still allocating logs. If you choose to let the machine do the allocation, and you happen to have a slow file system and disk speed, then annoying things happen. Normally this will not be a problem, but when it does, you can turn off pre-allocation by using undocumented Flag — nopreallocj.
Summary: If the machine file system and disk are too slow, the pre-allocation of logs can be slow.
NUMA + Linux +MongoDB
Linux, NUMA, and MongoDB don’t always work well together. If you are running MongoDB on NUMA hardware, it is recommended to turn it off. Because all sorts of strange things happen, like the speed stages or drops off dramatically when CPU usage is high.
Summary: Disable NUMA.
Process restrictions in Linux
If you experience a SEGMENTATION FAULT error when MongoDB is not fully loaded, you may find that it is due to the use of low or default open file or user process limits. 10Gen recommends setting the limit at 4K+, however the size of the setting depends on the situation. Read ulimit to learn more.
Summary: Add soft or hard open file or user process restrictions to MongoDB on Linux for a long time.
MongoDB Gotchas & How To Avoid Them