Hadoop is a technology ecosystem, and ZooKeeper is a very important technology in the Hadoop ecosystem.

When I studied and learned hadoop related technologies, I was very confused by two pieces of knowledge: hbase and ZooKeeper.

Hbase was confused because it was overturning my understanding of database modeling, while ZooKeeper was confused because I couldn’t understand what it was about.

An architectural design overview of remote invocation services

First of all, we need to further understand why a remote call service is needed in the application software service, what problems does remote call service solve in software design, and what theoretical basis is there for its architectural design?

Version 3.0 of the web Architecture brings with it a new web architecture overview, as shown below:

With remote invocation services, we can do business level clustering.

For example, a manufacturing enterprise generally has procurement business, production business, sales business and financial business. Traditionally, we develop a system for each business. If we reference remote invocation services, we can make these businesses into independent services. These services form a business cluster, and these services all use a unified remote invocation service as the entry point of operation.

In other words, no matter what kind of service is unified for callers, the front-end callers can achieve the unification of applications. Taobao.com is the most typical representative of the so-called unification of applications. We can operate various applications in the same website. It doesn’t happen that because the application is different we have to go back to a new address or log back into another system to do something else.

On the server side, it can completely get rid of the traditional development of client and server coupling, enhancing the professionalism and stability of the whole server side, so that the scalability and maintainability of the server side are easier.

If the servers also need to call each other, they can also call the service remotely. Because of the uniformity of the remote call service, it avoids the disunity of message and call mode between service calls and standardizes the whole development process.

If remote invocation of services has load balancing capabilities, the entire service cluster becomes a private cloud, so it is not an exaggeration to say that remote invocation of services is an important part of cloud computing.

What is the theoretical basis of remote Service invocation? This question may be a little problematic. In fact, WHAT I want to talk about is that the technical prototype of remote Service invocation is SOA (Service-oriented Architecture).

Before in cloud computing, SOA once is a hot spot in the technology of IT, even though many people say China SOA after doing a little bad, just like early DHTML, more criticism than praise, writing this article I dongli search in Beijing under the SOA, from the date of publication of the books and books evaluation number can see SOA has been somewhat neglected desolate.

Let me briefly introduce SOA, the definition of SOA:

SOA is a software architecture that contains four key concepts: application front ends, services, service libraries, and service buses. A service contains a contract, one or more interfaces, and an implementation.

The application front end can be thought of as the caller and the front end system I described above, the service library can be thought of as a cluster of services, and what is the other service here?

A service is a contract between a caller and a service provider to complete a particular business, in other words, encapsulated business rules.

, for example, we go shopping in taobao, order, payment, logistics and determine the payment of these operations on the server can have independent service provider, but from shopping to understand the concept, the independent service to constitute the complete shopping behavior, if you have any place out of the question, which will have corresponding different operation, Then this is definitely not a matter of the caller simply calling the service interface. It requires a higher level of business encapsulation to encapsulate the above operations into a unified service, which is called a service.

The last element, the service bus, is the main topic of this article: remote invocation of services.

The purpose of talking about SOA here is to remind people of the role of SOA, so that those who want to learn more about remote invocation services can understand remote invocation services from the perspective of SOA, and those who still don’t understand what remote invocation is can understand remote invocation services from the concept of SOA.

Remote invocation of services in detail

Remote call service technical explanation, explanation, blare ~ ~, the two words have a lot of pressure, I am afraid that I will have children’s shoes to see this title will think I’m going to write a complete set of technical implementation scheme, and the difficulty is too high, tens of thousands of words written estimate all say not clear, and really write so meticulous, estimates that a lot of people don’t understand the (hey hey, I’ve had no technical implementation, These are ideas, ideas), so detail is detail principle.

Now I put in the architecture diagram of the previous article, and let’s take a closer look at this diagram:

Traditional service invocation is the direct invocation between service providers and service callers. We can see from the architecture diagram that there is a remote call management component here. The remote call management component is an independent service system, in order to ensure the stability of the system.

It must also be a distributed system, but the distributed system and Web distributed system is completely different distributed system, the traditional Web application cluster is based on the HTTP protocol stateless characteristics of the design.

Because each HTTP request is an independent transaction, there is no relationship between different requests, so we can deploy Web applications to different servers. No matter which server the request is sent to, the corresponding services can be provided to users normally. However, the Session mechanism of Web applications is stateful. Therefore, traditional Web clusters require the operation of session synchronization, while large websites tend to abstract the session function into an independent cache system.

But here’s the remote call management component cluster principle, or principle is different from the Web application distributed cluster, the principle of distributed remote call management components can treat as a registry, it will record the caller information service providers and service, and will push the information to the service provider or service the caller, in order to guarantee the execution efficiency of system, The registration information is recorded in memory.

Imagine that if these registrations were lost, the entire system would become unusable, so a cluster of remote invocation management components is a cluster that guarantees data reliability and robust service delivery, rather than one built on the stateless nature of HTTP.

We here under the hypothetical cluster of remote call service running scenes, if we have 5 servers as a remote call service running server, then each server must have a registered information redundancy backup, when service malfunctioned one server, the failure of the data on the server will not be lost, In addition, the cluster should have a fault detection mechanism. When a server is found to be unavailable, the server can be removed in time. Zookeeper is the technical framework to solve this problem.

In addition to ensuring the stability and availability of the system, the data storage mode of the cluster is also very important. I mentioned earlier that the data storage of the cluster should have a redundancy mechanism. Besides the redundancy mechanism, there should be a data model suitable for fast access and read and write, and ZooKeeper just contains this data model. Therefore, the remote call service I designed is a very suitable scenario for ZooKeeper application.

The remote invocation manager component also has a heartbeat mechanism that checks the health and availability of the service provider.

The registration information contains the IP address and port number of the server. The remote call management component starts a thread to ping the IP address and port number of the application according to the timing.

If the remote call manager is not available, the number of times it tries again and how often it detects the heartbeat can be configured. If it fails after several attempts, the service is considered unavailable.

The caller is the active party, while the provider is the passive party. This is just like when you visit a website, if you are sick and do not go to visit, there is no need for the system to check whether you have been sick.

The remote call framework requires serialization and deserialization techniques, which many children do not understand, and the reason for this is that they do not understand the serialization and deserialization techniques.

Serialization technology is mainly applied to data persistence (data stored on hard disks) or network communication. Whether data is stored on hard disks or network communication, these data will be converted into binary data. Serialization is to convert the running object into binary data that can be stored and transmitted.

Deserialization can reverse the binary data into the original object information, and the restored object can still be operated by the program. And the remote call framework we designed is the program code that can be used between different systems, so we need to use the technology of serialization and deserialization.

There is a problem here, for example, when we transfer an object, the object is an inherited subclass of N classes, and the object may refer to other objects such as String, ArrayList, etc.

: so, in order to let the deserialized object serialization will these information is included in the binary data, and these information together for network transmission, resulting in extremely large amount of data transmission, and the JDK’s own serialization mechanism will lead to the attached information is bigger, so it is necessary to use better than JDK serialization mechanism, make the data quantity is small, And serialization and deserialization are more efficient.

For service providers and service callers I will provide a JAR package that each project will import, along with a configuration file to define some user-defined parameters.

For example, we use a configuration file named ycdy_config.properties, which contains the following key values:

  • Config_center_url=ip:port; This is to configure the REMOTE manager IP address and port number;
  • Server_type = the provider/consumer; Configure to be a service caller or provider, default to provider if not configured;
  • Provider_post=9999; Provider_post=9999; Provider_post=9999; Provider_post=9999;
  • Provider_session_timeout = 9000; The timeout of the service provider. If the actual invocation exceeds this timeout, then the service invocation timeout applies to the service caller and the provider is invalid
  • Tick_time=3000; Heartbeat time, which is the interval at which the remote call center detects the heartbeat of the server, for the service provider;
  • Again_time=3; When the service provider is unavailable, the heartbeat repeatedly detects the number of times. If the number exceeds the number, the service is marked as unavailable. Provider_session_timeout, Tick_time and Again_time are related to each other. This relationship should realize the specific control.
  • Ip_include_pattern=172\.17\.138.| 192\168 \. \ 0..This applies to service providers, because a server may have multiple IP addresses. When the remote calling service component receives the PROVIDER’s IP address, this configuration item is used to identify which IP address is available, in the form of regular expression.
  • Ip_exclude_pattern= IP addresses to be ignored for service providers;
  • Consumer_policy = random/rotate; There are only two load balancing strategies that I’m familiar with that apply to the caller, that the caller requests to the provider: one is using random numbers, and the other is polling, so that’s the two options right now;
  • Monitor_log=true/false; Whether to enable the monitoring log is applicable to the service provider. Any system log is the most important. Otherwise, production problems cannot be checked.

As you can see, the configuration files used by both service providers and service callers are consistent, and an application can be configured to be either a service caller or a service provider, which is very flexible.

Netty is a very good choice. When it comes to communication, it is a complicated topic. If I have time later, I will introduce it in detail. However, the IP address and port number of communication need to be pushed by remote call management component.

So how does a remote call service work in an application?

Let’s look at the spring configuration for the service caller and service provider, as follows:

<! - service provider configuration - > < bean id = "serverProvider" class = "cn.com.sharpxiajun.RmifSpringProviderBean" > < property name="interfaceName" value="cn.com.ITest"></property><! Property name="target" ref="clsTest"></property><! -- clsTest implementation ITest implementation class, clsTest here is a bean id value --> </bean> <! - service caller configuration - > < bean id = "clientConsumer" class = "cn.com.sharpxiajun.RmifSpringConsumerBean" > < property name="interfaceName" value="cn.com.ITest"></property><! <property name="serialType" value="hessian"></property><! Property name="compressEnabled" value="true"></property><! </bean>Copy the code

We find that this new configuration is different from the previous one, and that this configuration will be more suitable for generation development.

We first look at the design of the serverProvider, this bean corresponding class is cn.com.sharpxiajun.RmifSpringProviderBean, a parameter is a interfaceName inside, namely the external interface of the provider.

Here I will use reflection to inject the interface into the RmifSpringProviderBean. The target is the concrete implementation object, which is the business object. Note that the interfaceName must be an interface, because the caller will translate it based on the interface.

ClientConsumer design, the class of the bean is cn.com.sharpxiajun.RmifSpringConsumerBean, where the interfaceName value value corresponding is the remote interfaces defined, Consistent with the provider’s interfaceName, when the provider’s data is transmitted to the caller, it is deserialized into operable objects based on the mutually agreed interface.

The serialType option is to select the serialization mechanism of the JDK, or to call the serialization mechanism of the JDK. By the way, external serializers are also included in the JAR, and the option of compressEnabled is used to enable or disable transmission compression.

When the caller invokes the provider service, the Netty program in the JAR will combine the pushed information (mainly IP, port) with the Beans configured by Spring to complete a service invocation.

Remote invocation of services in detail

There is a heartbeat mechanism detection between the remote call management component and the service provider to check whether the service provider can properly provide services externally.

The detection method I proposed in the previous article is to use the ping method to detect IP and port numbers. In fact, there can be another method in the concrete implementation.

Because the service provider will introduce the JAR package provided by the remote call service, in this JAR package actually contains a heartbeat test interface, the remote call management component uses the same communication mode between the service provider and the service caller to call this heartbeat test interface, if the interface can be dialed, then the heartbeat test is successful;

If you find a callback impassability then repeated the test several times, if still not understand, to mark the service provider’s this server is not available, this method of heartbeat detection is superior to use ping, because the heartbeat detection is completely simulate the service provider and the caller’s communications, more in line with the results of inspection and the real situation.

When the heartbeat mechanism to detect the service provider some servers malfunctioned, should we also need to have a mechanism, timely notify the operations staff or corresponding developers, accomplish fault timely processing, so the remote invocation framework should also include a monitoring system health monitoring module, if your company have a special monitoring system so much the better.

There is also a push message relationship between the remote call manager component and the service caller, which I describe here.

When service providers start to send its own IP address and port number, the remote call management component receives the information then put the IP and port number to service the caller, but the lack of details, do not stand up to scrutiny, do a Web development of children’s shoes all know that want to access an application in addition to the IP address and port number, also need the name of the application, In my configuration file, I only push the IP and port number. There is no application flag to determine which application service is called. How can the caller determine which application service is called?

For service providers don’t really need to apply name, only need to provide the IP address and port number to go, because it doesn’t matter if you’re that kind of service, for the caller to see is a remote call service name, so the remote call management components need to service the caller only push IP address and port number, and the service name is the name of the remote call service, Which service is invoked by the service caller is the interface in the Spring configuration, which is the id of the specific service.

There is another detail I haven’t covered. How does the remote call management component know which services the service caller needs to invoke?

Here I want for a remote call management component development a set of Web application of it, and at last in the configuration file and a service type of marking, the tag only suitable for the service provider, when service providers start to push together into the remote invocation management component, and an important function of Web application, The remote call management component can know which services the service caller needs to invoke by comparing the service caller’s IP address to these tags.

Now that the relationship between the service caller and the service provider is established, how does the remote call management component push service information to the service caller?

We find that the relationship between the remote call management component and the service caller is a typical publish-subscribe relationship. The publish-subscribe pattern has a solution in the design pattern: the observer pattern. Here I briefly introduce the observer pattern:

The observer pattern defines a one-to-many dependency that allows multiple observer objects to listen to a topic object at the same time and notify all observer objects when the state of the topic object changes so that they can update themselves automatically.

Through the definition of observer mode, we find that the remote call management component is the subject object, and the service caller is the observer. In fact, the information push mode in the program is realized by using the observer mode.

In the previous article, I showed you the Spring configuration files for the service provider and the service caller. Then I showed you that the provider and caller can communicate with each other using the RmifSpringProviderBean and RmifSpringConsumerBean. I thought I was clear at first, that there was code and there was truth, but then I had a kid talking to me privately and asking how the hell does this work? On second thought, if this question is not explained in detail, there are a lot of people do not understand.

Here I use proxy mode, here is a brief introduction to proxy mode:

The Proxy pattern and Proxy provide a Proxy for other objects to control access to this object. In some cases, an object is inappropriate or cannot directly reference another object, and a proxy object can act as an intermediary between a client and a target object.

A core design of remote invocation of services is to ensure that the invocation rules between service providers and service callers are consistent, but the specific business processing can be varied. This scenario is a typical proxy pattern.

Proxy mode in the remote call service is not a common agency model, because this kind of the establishment of the agency relationship is carried out in the program is running, so we must choose to use reflection mechanism to realize the dynamic proxy mode, after reading the above quote, we found it, I use interface in the spring, This shows that my solution is to use the JDK’s own proxy mode, but using cglib’s dynamic proxy, using the implementation class to implement the proxy I feel that the design may be more flexible.

Development in the framework of the remote call service, there are two important technologies: thread and communication, the two techniques is the core of the remote invocation framework technology is difficult, because the thread and communication are complex, so this article does not expand their concrete implementation approach, but talk about their role in the remote invocation framework.

Before I talk about the role, I want to make an interject. In the last article, I talked about the remote invocation service framework as a service bus (ESB) in the HOT SOA technology of a few years ago. For those of you who have been exposed to this advanced technology, the service bus at that time was called ESB bus for short. While the communication medium of ESB bus is almost always WebService, SOA was derived from enterprise-level solution.

The remote call service I design now is for the application of the Internet. The remote call framework of the Internet does not use WebService but uses new communication media.

Netty, for example, or Mina, Apache’s top-level project. In fact, currently WebService is also an obsolete technology, which is eliminated because of its low efficiency, whether it is the size of the data transmitted or the efficiency of the request response.

The remote invocation of the Internet service is a high performance framework, its performance is better, and it contains all the characteristics of the original ESB bus, thus the technology of the Internet far leading enterprise applications, enterprise application relative to the Internet enterprise is the traditional handicraft industry, and the present application of Internet technology has already begun to lead the enterprise, This is the truth that backwardness will be beaten, there is no sense of innovation will be revolutionized by others.

Since we want to design a high performance remote call service framework, one of the indicators of high performance is high concurrency, and high concurrency refers to how many threads your service can open, so thread write is directly related to the availability of remote call service.

There are a few things to be aware of when using threads:

First, the kernel of the server is generally multi-core, so when writing threads to make use of these cores, a large number of threads to use then must use the pooling technology.

The Executor framework is very well designed and compatible with Netty. Finally, there is the issue of thread safety. The most difficult one is thread safety, and I have the least confidence in thread safety if I implement threads myself.

Another indicator of high performance is network communication, and here I’m using Netty, a Java open source framework provided by JBOSS.

Netty provides an asynchronous, event-driven network application framework and tools to rapidly develop high-performance and reliable network server and client programs.

Netty has many advantages, but its biggest advantage is Netty’s long connection and event-driven, Netty can help us shield many socket difficult to control details, its long connection design is very good, reduce the cost of TCP on and off, and event-driven mode greatly reduces the difficulty of development. But Netty itself has a drawback. The drawback is that Netty is an asynchronous communication framework.

But we saw we call scenarios are synchronized, so we must put an asynchronous request to a synchronous request, in front of the distributed web site architecture, I use this technique, ah, it’s a pity that at that time, use time is to turn asynchronous synchronous details I’m still not very clear, this is also I need to study, after first reserve down here, I’ll blog about it later when I understand.

Well, that concludes the architectural design details of the remote invocation service, and it’s time to move on to ZooKeeper, the other topic of this article.

Zookeeper technology description

In the remote call service, ZooKeeper is used in the remote call management component, and the service caller is the client of ZooKeeper. The remote call management component is also the core of the remote call service.

If the remote call management components at run time to hang out, then the application will be unavailable, so whether the remote call management components must be reliable, the reliability requirement and even higher than the level of the reliability of the service provider and service the caller, thus the remote call management component design must be distributed, and must be reliable distributed.

Remote call management component is an application completely in line with the ZooKeeper scenario or a standard ZooKeeper application. In order to facilitate the subsequent discussion, I will further explain the functions of remote call management component.

From my previous description, we know that the core function of the remote call management component is to store the configuration information of the communication between the service provider and the service caller, such as the IP address and port of the service provider, record the service category of the service provider, It also records the IP address and port number of the service caller (this information is entered in the Web management system that remotely invokes the management component) and the comparison between the service caller and the service provider.

For the service provider, the remote call manager component also provides a heartbeat mechanism to detect the health status of the service provider. If the remote call manager component finds that some of the service provider’s servers are faulty, it will update the configuration information of the service provider in time and push the changes to the service caller in time.

As mentioned above, from the perspective of configuration information storage, the remote call management component is actually a system that stores configuration information remotely, while the heartbeat mechanism and push mechanism are an observer mode, and the above functions are all functions in a distributed environment, requiring high reliability.

At the beginning of this article, I mentioned that ZooKeeper was one of the most confusing technologies in the Hadoop technology ecosystem. This curiosity is one of the main reasons I’ve been paying attention to ZooKeeper, and now I kind of understand what ZooKeeper is. Because every technology is profound and endless, especially good technology.

The most typical application of Zookeeper is that it can provide configuration services for distributed applications. To be specific, the configuration files we usually write need to be completed by an independent system in a distributed system, and the configuration services are dynamic.

Since ZooKeeper can provide distributed configuration services, we can understand the functions of ZooKeeper in the reverse direction based on the characteristics of distributed configuration services.

I am a Java engineer. When I do Java Web development, I will use a large number of configuration files. Generally, these files are completed by using properties property files.

This configuration information has several characteristics:

Property files are generally not large (here refers to the system running related configuration files, we do not take the internationalization of the understanding), configuration information is persistent, when used is first loaded in memory, read from memory.

Zookeeper can also do this, and its features are almost the same as traditional configuration files. Zookeeper has a file system, which is used to store small files. When we read configuration information, we read it in memory, which is very efficient. When writing information, ZooKeeper persists the configuration information.

That’s why there are books about ZooKeeper’s performance:

Zookeeper’s benchmark throughput can exceed 10,000 operations, and is several times higher for read-dominated workloads.

This sentence is very reasonable, small file writing speed is very fast, 10000 operations have no problem, are through memory, several times higher is taken for granted.

Configuration information of the system operation reliability requirement is high, since we are now using distributed system to complete the configuration information to read and write operations, so it is very important to ensure the accuracy of the information literacy, especially in writing, the request is either success or failure, the scene is estimated that many people think a see is this thread safe?

Yes, thread safe, can think of thread safety is cattle oh), but we want to see, we are now distributed system, the operation between different servers rather than the category of thread process category, so here just need new technology to assure the safety operation, in other words, process safety mechanisms.

In addition, the distributed configuration services are using distributed in order to guarantee the stability and safety of configuration services system, in order to continue to provide users with high quality service, these two seemingly unrelated problem, but there is a solution to solve this problem, the solution is a zookeeper Zab agreement.

To illustrate the Zab protocol, let’s take an example. For example, we use five servers as ZooKeeper servers. We send instructions to the ZooKeeper cluster, which are read or write operations.

Operate a:

Leader election: When ZooKeeper starts, these 5 servers will elect a leader machine and the other machines will be followers. If more than half of the followers have communicated with the leader and confirmed the status, this stage will be completed.

If the leader is healthy all the time, the operation of the leadership election will not be promoted. If the leader is in trouble, ZooKeeper will promote the operation of the leadership election again.

Here is a question I am not sure, is the leader of ZooKeeper to check the health of followers, should also use the heartbeat mechanism? If anyone knows, you can comment on it.

Action 2:

If the command is a write request, all write requests will be forwarded to the leader, who will broadcast the update to the followers. When more than half of the followers persist the change, the leader will submit the update, and then the client will receive a successful update response. This means of reaching consensus is designed to be atomic, and the operation either succeeds or fails.

The above operations ensure atomicity of read and write, no dirty data, and repeated election of leaders to ensure the reliability of the service.

Of course, there is a problem here, what if the leader fails? In this case, the ZooKeeper cluster repeats the previous leader election operation. This also explains why the ZooKeeper cluster requires an odd number of servers. If two of the five servers fail, the service can still run normally. If six servers fail, only two servers can fail, because if three fail, the remaining servers are not more than half, then ZooKeeper itself fails. So odd-numbered servers do not waste server resources.

For read operations, any ZooKeeper server can directly to the service, with few other operations, so efficient; Zookeeper updates the data in the memory only after all servers persist the data. Therefore, the write operation is much slower than the read operation.

The operation mode of Zookeeper data storage is the same as that of Unix file system. The memory data storage model is a tree structure. The nodes in the tree structure are called Znode, which is used to store and read data.

The configuration information we store is done using these operations.

For example, when the service provider starts to push its configuration information to the remote call management component, the component will create nodes or set the data saved by ZNode. When the data is saved successfully, ZooKeeper will immediately push the information to the service caller, and zooKeeper can also complete the push work.

Znodes in ZooKeeper have been changed in some way. Each ZNode has an observer mechanism attached to it, which is the application of the observer mode mentioned earlier. This observer mechanism notifies the client, which is the service caller.

If the heartbeat mechanism detects that a server of the service provider is faulty, ZooKeeper will also modify the corresponding ZNode information, which also triggers the observation mechanism to notify the service caller of the change.

All right, that’s it for this article.

Zookeeper has very powerful functions, not only including the application I mentioned above, this article is the beginning of my in-depth study of Zookeeper, with the current foundation, further study of Zookeeper will be much easier.

I’ve compiled the interview questions and answers in PDF files, as well as a set of learning materials covering, but not limited to, the Java Virtual Machine, the Spring framework, Java threads, data structures, design patterns and more.

Follow the public account “Java Circle” for information, as well as quality articles delivered daily.