Tiktok, Baidu, Aliyun and Tencent are all self-built data centers with t-level egress bandwidth (total egress bandwidth), that is, the egress bandwidth of 1T= 1,024g /s. The general administration of servers has more than 200,000 units, and even Aliyun has more than 1 million units.

Bytedance’s total data center bandwidth is likely to be around 10TB, with 15TB not far off.

Generally, the total egress bandwidth is 1TB, but the actual egress bandwidth of the equipment room may only be about 100 gb. Dual-link (multi-link) design is adopted to implement dynamic traffic balancing. The total egress bandwidth can reach T level.

To have hundreds of millions of people online at the same time, terabyte bandwidth, CDN acceleration and multi-node, load-balancing technologies are all needed. (This design technology is too complex, there are related professional friends, can comment on the brief overview)

How many servers does Bytedance have?

According to network data collation, there were 20,000-30,000 servers in 2017, which were mainly rented servers at this time. In 2018, Bytedance built its own data center, the largest of which is in the Guantinghu New Media Industrial Park in Huailai, Hebei province, with 50,000 servers in the first phase. Phase II 90,000 servers.

In 2018, the number of leased + self-built servers reached 170,000 servers.

In 2020, according to byteDance’s job announcement, there were 420,000 servers. That’s 1.5 times more than in 2018. (Data from the Internet)

This part of the server is for use in China, mainly using domestic products such as Douyin, Watermelon Video, Toutiao, Feishu and so on.

TIKTOK is a separate operation in the United States, where data is stored and distributed locally.

TIktok also rented nearly 100,000 servers in the U.S. in 2020

Bytedance leased a 53 megawatt data center in Northern Virginia in the first half of 2020, according to Business Insider. It can house hundreds of thousands of servers and cover hundreds of thousands of square feet.

Tiktok is investing in data centers in India and Singapore.

So with server counts out of the way, let’s get some hardcore stuff: What’s the outbound bandwidth for ByteDance’s big data centers?

The so-called export bandwidth is actually what we ordinary people call download bandwidth. That’s the total speed at which the server distributes data to each phone.

In general, small IDC companies build their own computer rooms, such as some website companies, rented unicom, mobile, telecom room, the overall export bandwidth may only be 5G. More than 30G are all enterprises with a certain scale. Network company revenue is at least tens of millions of enterprises.

Therefore, it is often seen that some of the scale of the enterprise is not bad, basically no longer build their own room, are the use of cloud host. For example, ECS of Ali Cloud, Tencent Cloud, Baidu Cloud and AWS (Amazon).

General an enterprise website (enterprise official website), 20M bandwidth, 4G memory, 100G hard disk, a year price of 4000-5000 yuan is enough, catch up to do activities may be cheaper.

This is the most expensive bandwidth, of course, increase bandwidth, to achieve a certain level, such as the increase in traffic, must increase memory and hard disk. By contrast, bandwidth increases, which is more expensive. Bandwidth is expensive and a scarce resource.

Let’s take a look at a machine room of China Mobile: Data from the Data center of China Mobile (Shijiazhuang, Hebei) :

It covers an area of 174 mu, with a total construction area of 130,000 square meters. It plans 10 single buildings, which can provide an installation capacity of about 30,000 racks after completion. Three IDC rooms provide 31,000 cabinets and 15 TB bandwidth.

A cabinet contains a maximum of 16 1U devices, a maximum of 12 2U devices, and four to seven 4 u devices.

We split the difference between a high-performance 2U and A 4U server (after all, mobile is too rich to buy low-end 1U devices). Then 31,000 cabinets can be installed, with a maximum of 210,000 to 360,000 servers.

Here’s a rough average: 300,000 servers. Enjoy 15 TB egress bandwidth resources. Of course, as the dry road network of telecommunications, mobile bandwidth resources must be more than Bytedance advantages.

So, we made a rough estimate of byteDance’s own data center of 170,000 servers. The total egress bandwidth may range from 7Tb to 10TB. Basically, it will adopt dual-outlet traffic design, coupled with multi-link deployment mode: the actual outlet bandwidth can be achieved at 800G-1TG to achieve the total outlet bandwidth of about 10T. [This is the technical way]

What is the concept of t-level egress bandwidth? If I tell you that in 2009, the egress bandwidth of Shanghai was only 1140G, that is, it just reached 1TB. In just 10 years, the eject bandwidth of an enterprise’s data center exceeded 1Tb, a speed that is really unimaginable.

In 2009, despite the lack of smartphones, PC sales were huge.

CDN acceleration, let the public brush Douyin, watch video are not card

CDN (Content Delivery Network)

The content of the server is published to the edge node closest to the user so that the user can get the content they need nearby.

Solve Internet network congestion, improve the response speed of users to access websites. A collection of accelerated schemes.

In popular words to explain CDN is: some pages, special compression, some compression for static pages, static pages directly distributed fast. The user can see the content within 2 seconds, which makes the experience better. [This is static transmission]

For dynamic video, intelligent routing is firstly used to find the best path, and then protocol optimization is used to compress long connections and content to remove redundancy. [This is dynamic compression]

Let me give you a look at the CDN level of Tencent 500 million daily activities in 2015, which integrates music, instant messaging and other products, reaching 10TB bandwidth. Trillions of requests per day.

So, I’m here to say that byteDance’s entire server has 10 terabytes, which should be a little less. After all, Douyin has 600 million daily users, and we can roughly count it as 200 million. In total, there are 800 million daily users.

It’s the bandwidth and technology that allows us to watch video so smoothly.

Author: Science and Engineering Notes