Finally, someone has spoken out about cloud computing, big data and ARTIFICIAL intelligence! (1)

This article has been authorized by the author Liu Chao netease cloud community.

Welcome to visit netease Cloud Community to learn more about Netease’s technical product operation experience.

Today I want to talk to you about cloud computing, big data and artificial intelligence. Why are we talking about these three things? Because these three things are very popular now, and they seem to be related to each other: when you talk about cloud computing, you talk about big data, when you talk about ARTIFICIAL intelligence, you talk about cloud computing… It feels like the three are mutually reinforcing and inseparable. However, if you are a non-technical person, it may be difficult to understand the relationship between the three, so it is necessary to explain.

First, the initial goal of cloud computing

Let’s start with cloud computing. The initial goal of cloud computing is to manage resources, mainly computing resources, network resources, and storage resources.

Managing a data center is like configuring a computer

What are computing, network, and storage resources?

For example, if you want to buy a laptop computer, do you care what kind of CPU it has? How much memory? These two are what we call computing resources.

To connect to the Internet, the computer needed a network port to plug in, or a wireless card to connect to our router. Your home also needs to open a network with operators such as China Unicom, China Mobile or China Telecom, such as 100M bandwidth. Then a master will bring a network cable to your home, and the master will probably help you configure your router to connect to their company’s network. All your computers, phones and tablets will be connected to the Internet through your router. This is the web resource.

You might also ask how big is the hard drive? Hard drives used to be small, like 10 gigabytes; Later, even 500GB, 1TB, 2TB hard disk is not new. (1 TB is 1000 GB), this is the storage resource.

That’s true for a computer, and it’s true for a data center. Imagine you have a very, very large computer room with a lot of servers. These servers also have CPU, memory, hard disk, and Internet access through routers. The question is: How do the people who run the data centers manage all of this in a unified way?

Flexibility means having everything you want and as much as you want

The goal of management is to achieve flexibility in two ways. What are the two aspects?

For example, can you give a person who needs a very small computer with only one CPU, 1 GB of memory, 10 GB of hard disk and 1 Megabyte of bandwidth? Like this so small specifications of the computer, now any notebook computer is stronger than this configuration, the home to pull a broadband to 100M. However, if he goes to a cloud computing platform, he only needs a little bit of this resource when he wants it.

In this case it is flexible in two ways:

• Time flexibility: You want it when you want it, and you get it when you need it.

• Spatial flexibility: As many as you want. Need a computer too small, can meet; Need a particularly large space such as cloud disk, cloud disk to everyone’s allocation of space is always very big, upload at any time there is space, never run out, but also can meet.

Spatial flexibility and temporal flexibility are what we call the elasticity of cloud computing. It took a long time to solve the problem of elasticity.

3 Physical devices are not flexible

The first stage is the physical device stage. The client needed a computer, so we bought one and put it in the data center.

Physical equipment is of course more and more cattle, such as the server, memory is always a hundred GIGABytes of memory; For network equipment, the bandwidth of a port can be tens or even hundreds of gigabytes. Storage, for example, is at least petabyte level in the data center (one P is 1000 T, one T is 1000 G).

However, physical devices are not very flexible:

• The first is its lack of time flexibility. You can’t get what you want when you want it. Buy a server for instance, buy a computer, want to have purchase time. If all of a sudden a user tells a cloud vendor that they want to turn on a computer and use a physical server, it’s hard to buy at that time. A good relationship with the supplier may take a week, and an average relationship with the supplier may require a month of procurement. Users wait a long time for their computers to arrive and then log in to deploy their applications. Time flexibility is very poor.

• Secondly, its spatial flexibility is also not good. For example, the above user needs a very, very small computer, but there is no such a small computer now? You can’t buy such a small machine just to satisfy users that only one gigabyte of memory is 80 gigabytes of hard disk. But if you buy a big computer, you’ll have to charge the user more because the computer is so big, but the user only needs to use so small, so it’s very unfair to pay more.

4 Virtualization is much more flexible

Somebody figured it out. The first is virtualization. Don’t users just want a very small computer? The physical devices in the data center are very powerful. I can virtualize a small piece of physical CPU, memory, and hard disk for customers, as well as for other customers. Each client sees only their own piece, but each client actually uses a small piece of the larger device.

The technology of virtualization makes the computers of different customers appear to be isolated. So I look like this disk is mine, and you look like this disk is yours, but maybe my 10 gigabytes and your 10 gigabytes are sitting on the same big, big memory. And if the physical equipment is ready beforehand, virtualization software can create a virtual computer very quickly, basically in a few minutes. So if you want to create a computer on any cloud, it will come out in a few minutes.

So spatial flexibility and temporal flexibility are basically solved.

5. Making money and feelings in virtual world

In the virtualization phase, the company that dominates is VMware. It is one of the early companies to implement virtualization technology, which can realize computing, network, storage virtualization. The company was great, did great performance, sold great virtualization software, made a lot of money, and was acquired by EMC.

But there are a lot of people with feelings in this world, especially among programmers. What does the person with feelings like to do? Open source.

A lot of software in the world is either closed source or open source, source is source code. In other words, if something is good, everyone likes to use it, but THE code of the software is sealed by me, and only my company knows about it, but no one else knows about it. If other people want to use the software, they have to pay me, which is called closed source.

But there are always some cases in the world where the big boys don’t like the fact that all the money goes to the family. They think, you know this technology, I know it; If you can develop it, so can I. I develop it is free of charge, take out the code to share with everyone, the world can use, all people can enjoy the benefits, this is called open source.

Tim Berners-Lee, for example, is a very sentimental man. In 2017, he received the Turing Award of the year 2016 for “inventing the World Wide Web, the first browser, and the basic protocols and algorithms that enabled it to expand.” The Turing prize is the Nobel Prize of computing. But what he is most admired for is that he gave the World Wide Web, the technology we all know as the WWW, to the world for free. Everything we do online today should be thanks to him. If he were to make money from this technology, he would be as rich as Bill Gates.

There are many examples of open and closed sources:

In the closed source world, for example, you have Windows, and everybody has to pay Microsoft to use Windows; In the open source world came Linux. Bill Gates made a lot of money from Windows, Office and other closed source software, and became the richest man in the world, so he developed another operating system, Linux. Many people may not have heard of Linux, many backstage servers running programs are Linux, for example, we enjoy double Eleven, whether taobao, JINGdong, Kaola… The systems that support the Double 11 shopping run on Linux.

If there is Apple, there is Android. Apple has a huge market cap, but the code for Apple’s operating system is invisible. So someone wrote the Android mobile operating system. So you can see that almost every other phone manufacturer has Android in it. The reason is that apple’s OS is not open source, whereas Android is available to everyone.

The same goes for virtualization software, which is very expensive with VMware. There are two open source virtualization software, one called Xen and one called KVM. If you don’t do technology, you can forget these two names, but I will mention them later.

Semi-automatic virtualization and fully automatic cloud computing

To say that virtualization software solves the flexibility problem is not entirely true. Because virtualization software generally creates a virtual computer, it is necessary to manually specify which physical machine this virtual computer is placed on. This process may also require more complex manual configuration. So the use of VMware virtualization software, need to test a very good certificate, and can get this certificate, salary is quite high, but also visible complexity.

Therefore, the cluster scale of physical machines managed only by virtualization software is not particularly large, generally in a dozen, dozens, at most a hundred such a scale.

This has an impact on time flexibility: although the time required to virtualize a computer is short, the manual configuration process becomes more complex and time-consuming as the cluster size increases. On the other hand, it also affects spatial flexibility: when the number of users is large, the cluster size is far from the desired extent, and it is very likely that this resource will be used up quickly and have to be purchased.

Therefore, as the scale of the cluster becomes larger and larger, it usually starts with thousands, often tens of thousands or even tens of millions. If you look at BAT, including netease, Google and Amazon, the number of servers is huge. So many machines have to rely on people to choose a place to put this virtual computer and do the corresponding configuration, it is almost impossible, still need the machine to do this job.

Various algorithms have been invented to do this. The name of the algorithm is Scheduler. Popular point said, is to have a dispatch center, thousands of machines in a pool, no matter how much the user needs to virtual computer, CPU, memory, hard disk scheduling would automatically in a big pool to find a place to meet user needs, the virtual computer boot up completes the configuration, the user can directly use. This phase is called pooling or cloud. At this stage, it can be called cloud computing. Before that, it can only be called virtualization.

7 Cloud computing private and public

There are roughly two kinds of cloud computing: private cloud and public cloud. Some people call the combination of private cloud and public cloud hybrid cloud.

• Private cloud: Virtualization and cloud-based software deployed in other people’s data centers. Private cloud users tend to have the money to buy their own land to build computer rooms, buy their own servers, and then let cloud vendors deploy in their own place. VMware has since launched cloud computing products as well as virtualization, and is making a fortune in the private cloud market.

• Public cloud: the virtualization and cloud software deployed in the cloud manufacturer’s own data center, users do not need a lot of investment, as long as the registration of an account, you can click on a web page to create a virtual computer. For example, AWS is Amazon’s public cloud; For example, Alibaba Cloud, Tencent Cloud, netease Cloud and so on.

Why would Amazon do a public cloud? As we all know, Amazon used to be a big e-commerce company in foreign countries, and it will certainly encounter a scene similar to double Eleven when doing e-commerce: everyone rushes to buy things at a certain moment. The time and space flexibility of the cloud is especially needed when people rush to buy things. Because it doesn’t have all the resources available at all times, and that’s a waste. But also can’t prepare anything, looking at double 11 so many users want to buy things on the board. Therefore, a large number of virtual computers are created to support e-commerce applications on Singles’ Day, and after singles’ Day these resources are freed up again for other purposes. So Amazon needs a cloud platform.

However, commercial virtualization software is too expensive for Amazon to give all the money it makes in e-commerce to virtualization vendors. So Amazon developed its own cloud-based software, based on open-source virtualization technologies like Xen or KVM, as described above. Unexpectedly, Amazon became more and more successful in e-commerce and cloud platform.

Because its cloud platform needs to support its own e-commerce applications; However, traditional cloud computing companies are mostly IT companies with few applications of their own, so Amazon’s cloud platform is more application-friendly and quickly develops into the first brand of cloud computing, making a lot of money.

Before Amazon announced its financial results for its cloud computing platform, people were speculating that if Amazon makes money on e-commerce, will the cloud also make money? Later a public financial statements, found not general money. Last year alone, Amazon’s AWS had annual revenue of $12.2 billion and operating profit of $3.1 billion.

8. Money making and feelings of cloud computing

Amazon, no. 1 in public cloud, is having a good time; Rackspace, no. 2, is having a so-so time. No way, this is the cruelty of the Internet industry, is the winner-takes-all model. So if the number two is not in the cloud computing industry, many people probably haven’t heard of it.

The second person thinks, what if I can’t beat the boss? Open source. As mentioned above, Although Amazon uses open source virtualization technology, the cloud code is closed source. A lot of companies that want to be a cloud platform but can’t do it just watch Amazon make a lot of money. Once Rackspace makes the source code public, the entire industry can work together to make the platform better and better.

As a result, Rackspace and NASA cooperated to create open source software OpenStack. As shown in the diagram above, the architecture diagram of OpenStack is not for cloud computing industry, but you can see three keywords: Compute, Networking, and Storage. It is also a cloud management platform for computing, network and storage.

Of course, the technology in the second place is also very good. After OpenStack, as Rackspace thought, all the big enterprises that want to do cloud have gone crazy. All the big IT enterprises that you can imagine, such as IBM, HP, Dell, Huawei and Lenovo, have gone crazy.

The original cloud platform we all want to do, looking at Amazon and VMware made so much money, looking at no way, want to do one seems to be quite difficult. Now, with OpenStack as an open source cloud platform, all IT vendors are joining the community, contributing to the cloud platform, packaging their own products and selling them with their own hardware. Some do private clouds, some do public clouds, OpenStack has become the de facto standard for open source cloud platforms.

9 IaaS: Flexibility at the resource level

As OpenStack becomes more and more mature, it can be managed on a larger scale and can be deployed in multiple OpenStack clusters. For example, Beijing deployable one set, Hangzhou deployed two sets, Guangzhou deployed one set, and then unified management. So the whole scale is bigger.

At this scale, for the perception of ordinary users, they can basically do what they want when they want, as much as they want. Or take the cloud disk as an example, each user cloud disk has been allocated 5T or more space, if there are 100 million people, it adds up to how much space ah.

Actually the mechanism behind is this: to distribute your space, you probably took only a few, such as it is assigned to the 5 T, you such a big space is only you can see, not really give you, you actually took only 50 G, is real to you is 50 G, as your files are uploaded, give your space will be more and more.

When everyone uploads and the cloud platform finds that it is nearly full (for example, 70% is used), it will purchase more servers and expand the resources behind them, which is transparent and invisible to users. In a sense, it achieves the elasticity of cloud computing. In fact, it’s a bit like a bank, which gives the customer the impression that they can withdraw their money whenever they want, so long as they don’t run at the same time, the bank won’t collapse.

10 summary

At this stage, cloud computing basically realized the flexibility of time and space; Elastic computing, network, and storage resources are realized. Computing, networking, and storage are often referred to as infrastructure Infranstracture, and therefore resiliency at this stage is called resource-level resiliency. A cloud platform that manages resources is called an infrastructure Service, or IaaS (Infranstracture As A Service).

Second, cloud computing is not only about resources, but also about applications

With IaaS, is it enough to achieve resource-level resilience? Obviously not. There’s also the elasticity of the application layer.

Here is an example: for example, to realize an e-commerce application, usually ten machines are enough, but singles’ Day needs 100 machines. You might think it’s easy, with IaaS, to create ninety new machines. However, the 90 machines were empty when they were created, and the e-commerce applications were not put on them. The operation and maintenance staff of the company could only handle them one by one, which took a long time to install.

Although resiliency is achieved at the resource level, it is not enough to be flexible without resiliency at the application level. Is there a way around this?

An additional layer is added to the IaaS Platform to manage the elasticity of applications over resources. This layer is commonly called Platform As A Service (PaaS). This layer is often difficult to understand and has two parts: one I call “your own app installs automatically” and the other I call “Generic apps don’t install”.

• Install your own apps automatically: For example, if you develop an e-commerce app, no one else knows how to install it but you. For e-commerce applications, alipay or wechat accounts need to be configured during installation, so that when others buy things on your e-commerce, the money paid is sent to your account, and no one knows except you. So the installation process platform doesn’t help, but it can help you automate it. You need to do some work to integrate your configuration information into the automated installation process. For example, the 90 machines newly created on Double Eleven are empty. If a tool can be provided to automatically install e-commerce applications on the new 90 machines, the real elasticity of the application layer can be realized. Puppet, Chef, Ansible, Cloud Foundary all do this, but the latest container technology, Docker, does it even better.

• Common applications do not need to be installed: The so-called common applications generally refer to some of the high complexity, but everyone is using, such as databases. Almost all applications use databases, but database software is standard, and although installation and maintenance are complex, it is the same regardless of who installs it. Such applications can become standard PaaS layer applications on the cloud platform interface. When the user needs a database, the point comes out and the user can use it directly. Some people ask, since who installs the same, that I come by myself, do not need to spend money to buy on the cloud platform. Of course not. Databases are a very difficult thing to do. Oracle alone makes so much money from databases. Oracle also costs a lot of money.

However, most cloud platforms offer open source databases like MySQL, which is open source, so you don’t need to spend that much money. However, to maintain this database, we need to recruit a large team. If this database can be optimized to support double eleven, it will not be able to be done every year or two.

Such as you do is a bike, of course, there’s no need to hire a very large database team to do this thing, the cost is too high, should be handed over to a cloud platform to do this, professional thing to do, cloud platform keeps hundreds of people to maintain the system, specialized application you just focus on your bike.

Either you deploy it automatically or you don’t deploy it at all, and in general you don’t have to worry about the application layer, and that’s where the PaaS layer comes in.

While scripting can solve your own application deployment problems, different environments vary, and a script that works correctly in one environment may not work correctly in another.

Containers are a better solution to this problem.

Container is another word for Container. The idea of a Container is to become a Container for software delivery. The characteristic of container: it is package, 2 it is standard.

In the days before containers, it would take three ports and three changes to get goods from A to B. Each time the goods were unloaded, laid in pieces, and then brought aboard to be put in order again. So when there were no containers, the crew had to stay ashore for several days each time they changed ships.

With the containers, all the cargo was packed together, and the containers were all the same size, so every time a ship changed, a single box was moved across the ship in one piece, which could be done by the hour, and the crew did not have to spend long time on shore.

This is the container “encapsulation”, “standard” two characteristics in life application.

So how does a container package an application? Still need to learn container. First of all, there should be a closed environment to encapsulate the goods, so that the goods do not interfere with each other, isolated from each other, so that loading and unloading is convenient. Fortunately, LXC technology in Ubuntu has long been able to do this.

There are two main technologies used in closed environments. One is a seemingly isolated technology called Namespace, which means that each application in a Namespace sees a different IP address, user space, procedure number, etc. The other is a segregated technique called Cgroups, which means that an application can only use a portion of the CPU or memory on the entire machine.

The so-called mirror is the moment when you weld the container, and the state of the container is preserved. Just like sun Wukong said, “Fixed”, the container is fixed at that moment, and then the state of this moment is saved into a series of documents. The format of these documents is standard, and anyone who sees them can reconstruct the moment of the moment. The process of restoring an image to runtime (that is, reading the image file to restore that moment in time) is the process of the container running.

With containers, the automatic deployment of the PaaS layer to users’ own applications becomes fast and elegant.

Big data embraces cloud computing

A complex common application in the PaaS layer is the big data platform. How does big data fit into cloud computing step by step?

1 Small data also contains wisdom

The big data wasn’t big at first. How much data was there before? Now we all go to read e-books, the Internet to read the news, in the 1980s when we were young, the information is not so big, also read books, read newspapers, a week of newspapers add up to how many words? If you are not in a big city, the library of an ordinary school adds up to few shelves. It is later with the arrival of information, information will be more and more.

First of all, let’s take a look at the data in big data, which can be divided into three types: structured data, unstructured data and semi-structured data.

• Structured data: data that has a fixed format and a finite length. For example, the form is structured data, nationality: People’s Republic of China, nationality: Han, gender: male, these are all called structured data.

• Unstructured data: Now there are more and more unstructured data, that is, data of variable length and no fixed format, such as web pages, sometimes very long, sometimes a few sentences disappear; For example, voice and video are unstructured data.

• Semi-structured data: Some XML or HTML format, non-technical people may not understand, but it doesn’t matter.

In fact, the data itself is not useful, must go through a certain amount of processing. For example, if you run every day and wear a bracelet, you collect Data, and so do so many pages on the Internet, which we call Data. Data by itself is not useful, but data contains something very important, which is called Information.

Data is messy, combed and cleaned before it can be called information. Information will contain many rules, we need to summarize the rules from information, called Knowledge, and Knowledge changes fate. There is a lot of information, but some people see the information is equivalent to nothing, but some people see the future of e-commerce from the information, some people see the future of live broadcasting, so they are cool. If you don’t extract knowledge from information, you can only be a spectator in the tide of the Internet by checking moments every day.

Some people will do very well if they have knowledge and then apply it to actual combat, which is called Intelligence. There is knowledge does not necessarily have wisdom, for example, many scholars are very knowledgeable, what has happened can be analyzed from various angles, but as soon as it comes to work, it can not be transformed into wisdom. Many entrepreneurs are great because they apply their knowledge to practice and end up doing great business.

So the application of data is divided into four steps: data, information, knowledge, wisdom.

The final stage is what many businesses want. You see, I have collected so much data. Can you help me make decisions based on these data to improve my products? For example, letting a user watch a video while an AD pops up next to the item he wants to buy. Another example is to ask the user to listen to music and recommend some other music that he really wants to listen to.

Users in my application or website randomly point the mouse, input text for me are data, I am to extract some things, guide practice, form wisdom, let users into my application inside inextricate themselves, on my network do not want to leave, hand kept location, kept buying.

A lot of people say double eleven I want to break the network, my wife in the above constantly buy buy buy, buy A and recommend B, wife adult said, “oh, B is also I like ah, husband I want to buy”. How do you think this program is so smart, so smart, and knows my wife better than I do, how does this thing work?

How can data be transformed into wisdom

There are several steps to the processing of data, and wisdom comes after completion.

The first step is called data collection. You have to have data, and data is collected in two ways:

• The first way is to grab, which is technically called grabbing or crawling. Search engines, for example, do this: they download all the information on the Web to their data centers, where you can search for it. For example, when you go to search, the result will be a list, why is this list in the search engine company? Because he took the data down, but you click on the link, click on the site is no longer a search engine company. For example, Sina has a news, you take Baidu search, when you do not point, that page in baidu data center, a little out of the web page is in sina’s data center.

• The second way is push, there are many terminals that can help me collect data. The Mi band, for example, can upload your daily running data, heartbeat data and sleep data to the data center.

The second step is data transmission. This is usually done in a queue because there is so much data that it has to be processed to be useful. The system can not be processed, had to queue up, slowly processing.

The third step is data storage. Data is now money, and to have data is to have money. How else would the website know what you want to buy? Just because it has data about your transactions in history, this information can’t be given to anyone else, it’s very valuable, so it needs to be stored.

The fourth step is data processing and analysis. The data stored above is the original data, the original data is mostly chaotic, there is a lot of garbage data in it, so it needs to be cleaned and filtered to get some high-quality data. For high-quality data, analysis can be carried out to classify the data, or discover the relationship between the data, and obtain knowledge.

Such as beer and diapers that wal-mart’s story, is through the analysis of people’s purchase data, discovered the man usually bought diapers, can buy beer at the same time, thus found the relationship between beer and diapers, acquire knowledge, and then applied to the practice, will get very close, beer and diapers counter received wisdom.

The fifth step is the retrieval and mining of data. Search is to search, the so-called foreign affairs do not ask Google, domestic affairs do not ask Baidu. Both domestic and foreign search engines put analyzed data into search engines, so when people want to find information, they can find it in one search.

The other is mining, just search out can not meet the requirements of people, but also need to dig out the mutual relationship from the information. For example, when searching for a company’s stock, should the company’s executives also be uncovered? If you just search for the company’s stock and find that it is rising very well, then you buy it. In fact, the senior executive makes a statement, which is very bad for the stock, and the next day it falls, doesn’t that hurt the majority of shareholders? Therefore, it is very important to mine the relationship in data and form knowledge base through various algorithms.

The basic cloud computing services of netease deeply integrate IaaS, PaaS and container technologies, provide elastic computing, DevOps tool chain and micro-service infrastructure services, help enterprises solve IT, architecture, operation and maintenance problems, and make enterprises more focused on business. As a new-generation cloud computing platform, you can click to try IT for free.

Free experience cloud security (EASY Shield) content security, verification code and other services

For more information about netease’s technology, products and operating experience, please click here.

Related articles: [recommended] two ways to make Docker images [recommended] netease object storage NOS Map bed artifact [recommended] Android applications to prevent being repackaged guide