DynamoDB is a NoSQL database service implemented by Amazon based on Dynamo: Amazon’s Highly Available Key-Value Store. It can meet the database seamless expansion, can ensure the persistence of data and high availability. Instead of having to worry about maintenance, scalability, performance, etc., Dynamodb is fully hosted by Amazon, allowing developers to focus more on the architecture and business level.

This paper mainly introduces the challenges encountered by the author’s team in the specific business, based on why these challenges were finally selected to use Amazon Dynamodb, what problems were encountered in practice and how to solve them. The technical details of Amazon Dynamodb will not be discussed in detail, nor will the full features of Amazon Dynamodb be covered in this article.

Background and Challenges

As an advertising monitoring platform between advertisers and media, TalkingData Ad Tracking needs to receive a large number of promotion sample information and actual effect information every day, and ultimately the actual effect is attributed to the promotion sample.

For example, when we browse information through some news APP on mobile phones, we will see advertisements interspersed in the information stream. The advertisements may be in the form of text, pictures or videos, and they can interact with users no matter which form they are.

If the AD is precise and just the content that the user is interested in, the user may click on the AD to learn more information. Once the advertisement is clicked, the monitoring platform will receive the click event triggered by the user. We call all the information carried by the click event as sample information, which may include the source of the clicked advertisement, the time of the clicked advertisement and so on. Usually, after clicking the advertisement, the user will be guided to carry out relevant operations, such as downloading the APP recommended by the advertisement. When the user downloads and opens the APP, the mobile advertising monitoring platform will receive the effect information sent by the APP. So far, for advertising to do even a successful transformation.

Dynamodb Practice: When the data volume is large and unpredictable, how to ensure high availability and real time?

Mobile advertising monitoring platform needs to receive a steady stream of sample information and effect information, and repeatedly, non-stop real-time processing and transformation time after time. For the monitoring platform, it has a great responsibility, can not record more, can not record less, if the converted data is recorded more advertisers need to pay more advertising fees to the media, the media will have a loss. This creates several challenges for the platform:

• Large amount of data: in order to maximize profits, some media may take abnormal means to create fake samples and generate “false traffic”. Therefore, in addition to receiving real user samples, advertising monitoring platforms will also receive a large number of fake samples, which will affect normal monitoring and attribution. At the height of our “craziness,” our platform received 4 billion + click sample event requests in a single day. Keep in mind that these click sample events are retained for subsequent effect attributions, and the sample validity can vary from a minimum of 12 hours to a maximum of 90 days.
• The amount of data is unpredictable: a series of promotions, such as a big push by advertisers or a bid ranking in the app store, will result in a sudden influx of large sample data. In the face of these unpredictable traffic, we still need to ensure that the system is correct, stable and real-time.
• Real-time processing: advertisers rely on the results of real-time processing by the advertising monitoring platform to adjust their advertising promotion strategies. Therefore, advertising monitoring platform needs to support real-time data processing in order to provide strong support for advertisers to optimize promotion strategies faster. At the same time, the results of advertising monitoring platform also should be transmitted back to the media and advertisers in real time. It can be seen that accuracy and real-time performance are the basic conditions that the system must meet.
• Sample storage: The most core function of our business is the attribution, we need to have a clear such as users to download to open the APP and which one is the transformation effect promotion sample – that is, from above in step 7, when users to install APP, monitoring platform to corresponding found in step 1 sample in promotional activities, this is a query matching process. For large attribution sample data, the validity period is different. How to store samples so that the system can make fast attribution without affecting real-time results is also a big challenge.

The original form

Before June 2017, our business processing service was deployed in the computer room and used Redis Version 2.8 to store all the sample data. Redis uses multi-node partitions, with each partition deployed in a master-slave manner. At the very beginning, we deployed multiple nodes in Redis, divided into multiple partitions, each with one master and one slave. There were no problems with this approach at the beginning, but as the validity period of samples set by users was extended and the number of monitoring samples increased, the number of nodes at that time gradually became insufficient to support the level of business storage. If the user monitoring promotion volume once exploded, our system storage will face collapse, the business will also be paralyzed. So we did the first expansion.

Due to the previous deployment we were only able to double the size of multiple Redis nodes, all of which had to be done manually, and in the meantime we did everything we could to protect the user’s sample data.

This type of deployment becomes more and more overburdened as the monitoring volume increases and the user set expiration period increases, which can have serious consequences when unexpected bursts occur. Moreover, manual expansion is error-prone, low timeliness, and the cost doubles. At that time, due to limited machine resources, not only Redis needed to expand capacity, but also a series of services and clusters of advertising monitoring platform needed to expand capacity.

Dissolve the challenge

After discussion and evaluation, we decided to migrate sample processing and other services to cloud processing, and re-selected the storage mode to Amazon Dynamodb, which can meet most of our business needs. After structural adjustment, the system will look like the following figure:



Dynamodb Practice: When the data volume is large and unpredictable, how to ensure high availability and real time?

• Coping with large and unpredictable data: Our platform needs to accept population-based monitoring connection requests and persist them for subsequent data attribution processing. Theoretically, Dynamodb can store as many AD monitoring data requests as come into the system. A single table can store any number of orders of magnitude of data. No need to worry about Dynamodb capacity expansion, we can’t perceive the storage capacity expansion when the system is running. This is Amazon’s official claim to be a fully hosted, seamless extension.
• High Availability: Amazon Dynamodb provides extremely high availability as a storage service. All data written to Dynamodb is stored on the SSD and automatically synchronized to AWS availability zones to achieve high availability. This work is also fully hosted by the Amazon Dynamodb service, allowing the consumer to focus on business architecture and coding.
• Real-time processing: Amazon Dynamodb provides extremely high throughput performance and supports configuring any level of throughput in a second dimension. For applications that write more and read less, you can adjust the number of data writes per second to 1000 or more, and the number of data reads per second to 10 or less. Throughput can be set arbitrarily by users. When setting throughput, it can not only be adjusted at any time in the background of Web management, but also be adjusted dynamically by the client provided by Dynamodb. For example, when the system is running, the writing capacity is insufficient. We can choose to manually adjust it to the Web management background or automatically adjust it by calling the client API in the code. The use of client dynamic adjustment will allow the system to have a higher shrinkage capacity, but also can ensure the real-time processing of data, the system data flow becomes high on the dynamic adjustment, data flow becomes low and then dynamic adjustment down. Compared to manual adjustment, dynamic adjustment is more flexible. Based on the above points, we believe that Amazon Dynamodb can easily support the core business capabilities of the system. For the business side, all you need to do is to sort out the business logic and write the data to Dynamodb, and then leave the rest to Dynamodb to do.

In addition:

• TTL: We take advantage of the TTL feature provided by Amazon Dynamodb to manage data that has a life cycle. Dynamodb will delete the expired data in the background once the timestamp has expired, similar to the concept of TTL in Redis. With the TTL capability, we eliminate many of the logic decisions that are not necessary for the business, and we also reduce the cost of storage.
• Streams: We do not enable streams to capture table actions in our business, but we think Dynamodb streams are a nice feature to notify the relevant services/programs when data stored in Dynamodb tables has changed (added, modified, deleted). For example, if we modify a field of a record, Dynamodb can capture the changes in that field and write the results before and after the changes into a stream record.

Real knowledge comes from practice

There are always some “pits” in the use of open source frameworks or services, which can also be interpreted as rules of use that are not well understood and addressed. Dynamodb, like all services, has its own set of rules. Here we mainly share in the actual use of the process encountered problems and solutions.

Data migration

When creating a table in Dynamodb, you need to specify the primary key of the table. This is for data uniqueness, fast indexing, and increased parallelism. There are two types of primary keys, “use partitioning key alone” as the primary key, and “use partitioning key + sort key” as the primary key, the latter can be understood as a combined primary key (index), which uniquely identifies/retrieves a piece of data by two fields. Dynamodb’s underlying database partitions the data based on the primary key values, so that it can be load-balanced to reduce the pressure on individual partitions. Dynamodb also tries to partition the primary key values “properly”.

We didn’t do anything with the primary key at the beginning, because Dynamodb will take the partitioning key value as the input to the internal hash function, and its output will determine which partition to store the data on. However, we found that the data started to be offset by writes, which was very serious. As a result, the performance of the DynamoDB table was degraded. The reasons will be discussed in detail later. After identifying such problems, we considered two solutions:

So we chose the second method, which is to adjust the business code to hash the primary key value when writing, and hash the primary key condition when querying.

Hidden rules for automatic capacity expansion

The read/write performance recovered after the data offset was resolved, but after running for a while the read/write performance deteriorated again. The data write is not offset. At that time, we improved the write performance to 60,000 +/ s, but it didn’t make any difference. The actual write speed was only 20,000 +/ s. The number of partitions maintained by Dynamodb in the background has reached 200+, which seriously affects the read and write performance of Dynamodb tables.

Dynamodb automatically scales and supports any throughput set by the user, based on its two automatic scaling rules: single-partition size limit and read/write performance limit.

Single partition size limit

Dynamodb maintains data storage partitions automatically, but each partition size is capped at 10GB, exceeding this limit will cause Dynamodb to de-partition. Dynamodb will secretly split your offset partition when the data is seriously offset. We can calculate the number of partitions using the following formula: total data size / 10GB rounded up = total number of partitions

For example, if the total data in the table is 15GB, 15/10 = 1.5, rounding up = 2, the number of partitions is 2, if the data is not offset evenly distributed, the two partitions each store 7.5GB of data.

Read and write performance limitations

Why did Dynamodb split the partition? This is because it ensures the user’s preset read/write performance. How do you guarantee that? Rely on each partition data control within 10G. Another condition is that Dynamodb will scale the partition when it cannot meet the preset throughput. Dynamodb defines read/write capacity for each partition as follows:

• Write capacity units (WCU: write capacity units), calculated at a maximum of 1KB per piece of data, the maximum number of writes per second is 1000.
• Read capacity units: RCU: Read capacity units, calculated at a maximum of 4KB per piece of data, with a maximum of 3000 pieces read per second.

In other words, a partition’s maximum write capacity and read capacity units are fixed, exceeding the partition’s maximum capacity units will be split. So we can calculate the number of partitions according to the following formula:

(preset read capacity /3000) + (preset write capacity /1000) and rounding up = total number of partitions

For example, the preset read capacity is 500, write capacity is 5000, (500/3000) + (5000/1000) = 5.1, and then rounded up = 6, the number of partitions is 6.

It should be noted that for single partition more than 10G split after the new partition is to share the original partition read and write capacity, not each form unique read and write capacity. The number of partitions is determined by the preset read and write capacity, but two new partitions are split due to the maximum data volume of a single partition. So when the data is heavily offset, read and write performance can degrade dramatically.

Hot and cold data

The above problem arises because we started with a single table operation. In this way, even if the data is not offset, but as the passage of time more and more data, natural split more and more partitions.

Therefore, we have done a reasonable table according to the business, set the hot and cold data table. This has two main benefits:

1. Performance improvement: This is obvious from the above rules. The amount of data in the heat meter does not continue to grow indefinitely, so the partitions are stable within a certain order of magnitude, ensuring read and write performance.
2. Cost reduction: Adding read and write capabilities to a single table is not only ineffective, but also dramatically increases the cost of use, which is unacceptable to anyone. Dynamodb storage also costs money, so it is possible to store cold table data in S3 or other persistent services. Deleting Dynamodb tables is another way to reduce the cost.

Table limit

There is no limit to the size or amount of data that can be written into a table. However, for an account at AWS, the limit for each Dynamodb usage zone is 256 tables. For a company, there is a risk that table creation will be limited if you share the same account. So if the hot and cold table policy is enabled, it is also a solution to the 256 table limit in addition to reducing the cost of cold table deletion.

Attribute name length

The limit of 1KB per write unit and 4KB per read unit was mentioned above. The size of a single piece of data occupies bytes in addition to field values, so attribute names in the table should be minimized to ensure readability.

conclusion

There are also costs associated with using DynamoDB, mainly in the form of write and read costs. We have developed a strategy to adjust read and write caps in real time based on actual traffic. With the development of auto-scaling, Dynamodb also introduced the function of auto-scaling, which realized the ability to dynamically adjust the write and read upper limit of custom strategy, and saved a lot of research and development energy for developers. At present, some of our businesses also use the function of Auto Scaling. However, due to the limitation of this function, the real-time performance of dynamic adjustment is somewhat lacking in actual use.

Author introduction: Shi Tianshu, senior Java engineer, graduated from Beijing University of Posts and Telecommunications with a master’s degree. Working at TalkingData, currently engaged in the architecture design and development of mobile advertising monitoring product Ad Tracking. Like to study the code, pay attention to the system high expansion design, a little code cleanliness.