preface

Recently wrote a lot of database related articles, we basically have a lot of understanding of the database, the database itself has some understanding, should we return to the business itself?

Have you learned how to deploy your enterprise database? How and where is it deployed? What are the possible problems during deployment?

Is the Lord? Double master? Are there any separate libraries? Have the big tables been divided? And so on… Deployment mode is also a large probability of sub-database, table order of magnitude more than ten million basically began to sub-table, considerate enterprise, certainly there are databases of cold backup, hot backup, disaster, and remote disaster recovery and so on.

I still remember my university project, the school is to buy a lot of physical machines, our project and database are deployed in their own internal server, that guy in the summer wind I hum, tired to death, the computer room is very hot.

But I would bet that most of you are currently working in an enterprise that uses the service deployment methods of various cloud service vendors, which brings us to our first thought.

Why do databases go to the cloud?

Most of our services and databases are in the corresponding cloud service vendors, then the question comes, why all on the cloud?

When thinking about this problem, I immediately thought of the contradiction method, not cloud harm is what?

  1. The cost of

Compared with traditional servers, which need to be purchased and rented, cloud servers adopt pay-as-you-go mode to reduce the purchase cost. The capacity of flexible expansion can be determined according to their own needs, and there is no need to estimate how much needs to be used in advance.

In the e-commerce activity team I worked for before, we would rent the data machine of cloud service manufacturer every time we came to the promotion, and we would return it after the activity, which really maximized the cost and still charged according to your traffic usage.

If we still use their own servers, then this time to temporary procurement? Although I know That Baidu bought more than one physical machine during the Spring Festival of one year, but the nature is different, they can maximize the use of these servers, they can even develop their own cloud services as a cloud service provider, in fact they did.

  1. performance

The cloud server realizes hardware isolation and broadband exclusivity. It is not limited by region and traffic, and can continue business communication without interruption.

If we still use the physical machine, the bandwidth cost of the dedicated line to the operator, and the performance of the physical machine may not be more.

Due to the current cost problem, your company has bought a lot of low-quality servers, but suddenly your business volume increases geometrically. What should we do? Keep buying the premium? Not a good fit, obviously. Who can handle that?

  1. management

The cloud server can realize remote synchronization management, sharing, and backup of various services. A traditional server must be in a certain network area and may be affected by the network, resulting in data loss.

The cold backup, hot backup and DISASTER backup mentioned above can be done by the server we buy, but it’s really a waste to put a server that we don’t know when it can be used.

And he can’t do, such as disaster, if your company in the quake zone, if still use physical server, basically is equal to the suicide, sending global users when natural disasters are unable to visit you, to the service vendor is different, their location is to have cultured, and set up their own data center in every place, guarantee the high availability.

  1. security

In order to ensure the reliability of cloud platform, cloud service platform companies will certainly invest a lot of effort, a reliable security guarantee system, platform users do not need to worry about the stability and security of the platform.

Once the owner of a physical machine with high authority breaks down, it is basically an unrecoverable disaster. Although cloud services are the same, reasonable use and proper authority convergence can achieve a higher level of security.

As we all know, if you do all kinds of full backup in advance, there is no big problem with incremental backup. Moreover, there is the problem of permission convergence. Our company disables extreme operations such as RM -RF, fdisk and DROP on the corresponding database server.

All database queries are their own component queries that cannot even be updated (only by code).

If you still use a physical machine, you need to maintain it by yourself, upgrade and patch it, which is difficult to ensure that it will not be invaded by hackers. I have encountered server patch late before, which led to hacker attack, hijacking and mining, while the security system of cloud service manufacturers is updated in real time.

Summary: There are no special circumstances, can use cloud products directly cloud products, because cloud products provide not only product capabilities, the most critical is the critical moment disaster recovery, emergency response and service capabilities, these capabilities, not all companies can build a complete set, or even many companies can not think of.

So far, although the major cloud vendors and their products have had some problems, the cloud is still the most complete and standardized system, and frankly, better than 99% of the companies.

Issues to consider on the cloud

This is very interesting. When I was writing this article, our company was engaged in some businesses such as cloud and cloud migration, which reminded me of many interesting things.

We are now migrating from a cloud to Huawei cloud, I think we will also with such a scenario, but this migration will bring some what kind of problems? I don’t know if you’ve thought about it?

In fact, from the local to the cloud, or from the cloud to the cloud, to think about the point is estimated to be much worse, then I first throw out some questions, see how huawei cloud service vendors solve these problems.

  1. Migration failure: What if data migration fails
  2. Data loss: How to determine whether data is complete after migration
  3. Business interruption: What about force Majeure halfway through migration
  4. Data and transmission encryption: how to encrypt data during transmission to prevent unauthorized persons from obtaining data
  5. Hot switching: how to achieve non-stop service switching, and data consistency in the process of switching data sources

We have to consider these problems, do you think migration is easy, then I would like to ask, if it is the order library? Bigger e-commerce companies place orders every second, or even every millisecond, even in the wee hours of the morning. Don’t ask me how I know.

You can’t stop migrating the database. You need to accept new data as you migrate.

rehash

In need of expansion, redis will create a new hash dictionary, this time the old stop receiving data, new data into a new dictionary, slowly get old data at the same time, this idea actually, also can be used in the database migration, but the operation of the database, are usually based on the data, not all is incremental.

That simple, do some tricks can also be done, the cloud manufacturer has put all the problems I mentioned above must have considered, I contact Huawei cloud, Huawei cloud uses DRS (Data Replication Service) to do the database migration, how did he do it?

DRS: Data Replication Service (DRS) is an easy-to-use, stable, and efficient cloud Service for online database migration and real-time database synchronization. DRS centers around a cloud database, reducing the complexity of data flow between databases, effectively helping you reduce the cost of data transfer.

You might wonder, why not do your own data migration, using someone else’s components? In fact, if you don’t have a better idea about the wheel, you can just use the ones written by others. Can you beat the research and development results of the professional team?

However, the implementation of the technology behind the solution of the problem or we need to care about, otherwise DRS everything for us to do, we just click the mouse to solve, how do you get harvest? That is the focus of today’s discussion.

Let me talk about the advantages of using wheels: lower costs, lower technical barriers and lower risks

  • Labor cost time cost, are very expensive, if a ready-made things for us to do, we also go to the development of why? Again, I’m sure most companies still don’t have a DBA, but now that the wheels are in, we can do things like migration, right? Our traditional technology transfer time and energy not to say, the failure rate is really high, and data comparison and so on, very headache, MY previous employer database migration is the middle of the night, do a night, dawn is not necessarily good, if not good, the user online, but also suspended.

But even with the use of tools, a complete database migration process should still be very rigorous, we may wonder how rigorous can be more rigorous? Here’s a picture:

Huawei cloud DRS online migration how to do it?

It can be seen that VP* is used in the migration diagram. The main function of this is to maintain a high-speed and stable transmission, as well as the encryption of transmitted data. In case your synchronization process is caught by other rival companies, then? At the end of the article, you can see how Huawei cloud DRS do network security, I did a complete migration of actual combat, and made a summary.

The migration of actual combat

His migration is very simple, there are tutorials, I used once, the general steps are as follows:

Migration as a special period, business cooperation and artificial cooperation are the most critical, and some operations must be avoided, such as common ones:

  • Source database logs cannot be forcibly cleaned up
  • The user password used to connect to the source database cannot be changed or deleted
  • The table cannot be locked for a long time. As a result, external users cannot query the table

He can do a pre-migration check before the migration. According to the official documents, they are all inspection steps summarized from previous migration cases, which can ensure the success of the migration. This is good.

Is no, you don’t know what to think of data migration, but if the database Settings didn’t migration that is also very troublesome, if a migration tool can set the DBA’s good User permissions migration, and all kinds of trigger, we set up the database character set migration, that is my ideal of a migration tool, Yes Huawei cloud DRS did, this is a better point, really save a lot of effort.

Especially for the database Settings are not so familiar with the development, this function is indeed very good, and there are performance parameters, such as a variety of buffer size, cache size, etc., he can migrate, and even flow control can be done, but also change the flow control is better:

Migration mode diversification, this is the first feeling I am ready to start migration, I mentioned above, if not incremental migration will be meaningless, DRS still thought of, this makes me feel like a little warm, said my eyes wet again…

Because most of the scenes are online business non-stop service migration, in the process of migration, there are still continuous incremental data in the influx, aobin experienced before the database migration is basically full + incremental migration mode, full scenes only exist in the internal system, or offline data.

In fact, the core of the technology here is how to ensure that the incremental data is not lost and the correct migration, I guess through binlog synchronization, I looked at his documents, logs, and I was right.

DRS migrates historical data to the target database through the full migration process. In the incremental migration phase, DRS captures logs and applies logs to ensure data consistency between the source database and the target database. DRS provides complete data comparison functions.

The migration process is simple, the progress can be seen completely, and the data delay can be seen intuitively:

After the migration, DRS provided data comparison. In fact, when I used to do data comparison, we always compared the number of rows in the database. Since there was no such migration tool, I found a warm point was content comparison, which surprised me because the comparison of the number of rows was not rigorous enough. Modified logs are useless if the row count comparison is missing.

After the comparison is completed, click “View comparison Report” to learn the details of the comparison. The details page is as shown in the figure below:

I also found the answer to the network security question mentioned above in DRS, they use a specific encryption protocol for data transmission, and can also use a specific VP* mount network transmission:

DRS do migration monitoring, can see real time schedule, make the whole migration progress is more visual, in the middle of the abnormal also be clear at a glance, to be honest tool is really sweet, dare not even think about before, we stay up late is lest a link error, and often still in hindsight, visualization of the process will do you have a sense of control on migration.

Migration completed:

From the beginning of my migration to the end, the whole process actually took less than 2 hours, which was unimaginable before. I was very satisfied with this wave of experience, which enabled me to do what only a DBA could do before in a single development. The corners of the eyes of the DBA beside me were wet….

summary

The overall experience was pretty good, and I’ve summarized a few of the design and usage scenarios that I think are unique to DRS:

  1. Migration rate limit: Sets the migration speed upper limit based on the specified period

Application Scenarios:

  • For some traffice-oriented apps, such as game manufacturers and other customers, the public network and VP of the source database cannot be filled up during migration (this will affect their external services or affect the sharing of VP bandwidth).

  • Some businesses are heavily loaded, or customers are unable to accept the extra load on business time applications due to migration

  1. User migration (permissions, passwords, definer) inherits the full source permissions system

Application Scenarios:

  • The migration product on the market are not support user migration, that is to say, if the user didn’t pay attention, or do not understand the user migration, then migrated business must be an error, DRS provides a complete set of user permissions inheritance design, can keep the permissions, passwords, definer migration to the target database, ensure the security of the migrated authority, business stability, It allows customers unfamiliar with the database to migrate while still completing a sophisticated, high-quality database migration.
  1. After the migration, services are stable

Application Scenarios:

  • The migration product on the market are not support parameters of migration, and the database parameter is different, this will directly lead to the business program is running an error (for example the session after several migration becomes), DRS selected business and related key parameters being strong of performance, to avoid the subsequent because there is no inheritance source environment parameters setting, and lead to business error or performance degradation, It allows customers unfamiliar with the database to migrate while still completing a sophisticated, high-quality database migration.
  1. Data proofreading platform, cutover good helper

Application Scenarios:

  • Data comparison is not supported by any of the migration products on the market. Calibration is left to user testing. DRS provides a variety of comparison functions:

  • Object contrast

  • Data-level comparison

  • Comparison can be timed and cancelled

    By using scheduled comparison tasks, you can perform data consistency comparison during off-peak hours, such as early morning, and view the data comparison results the next day to fully understand the migration status. It allows customers unfamiliar with the database to migrate while still completing a sophisticated, high-quality database migration.

  1. Migration of triggers and events

Application Scenarios:

  • The migration products on the market do not support the migration of triggers and events. Users who are proficient in migration pay attention to these details, because triggers and events are also part of the database, and there are key business logic in triggers and events. If these objects do not support migration, business will inevitably report errors and even cause irreparable losses.
  • It allows customers unfamiliar with the database to migrate while still completing a sophisticated, high-quality database migration.

Note: [Some of the pictures were deleted from the Internet]

conclusion

In fact, I would like to introduce a background and technology of DRS to you, mainly hoping to do a complete data migration with you and me, and discuss the scenes behind the technology together, and we should have technical thinking behind the scenes.

However, this experience really made me sigh with the convenience of technology. In the past, database migration was all team development and testing. A team stayed up to watch the database migration and finally verified the test.

Now I have finished a large-scale database migration exercise just by looking at the tutorial. While enjoying the convenience brought by the technology, I have also been deeply thinking about the concrete realization behind the technology and the meaning of life.

Maybe this is the value of technology, maybe this is the meaning of so many engineers working hard day and night, maybe…

I’m Aobing, and the more you know, the more you don’t know, and I’ll see you next time!