preface

With the popularity of microservices today, distributed systems are becoming more and more important. To realize servitization, the communication between services must be considered first. This involves serialization, deserialization, addressing, linking, and so on. However, with the RPC framework, we don’t have to fret.

What is RPC?

Remote Procedure Call (RPC) is a computer communication protocol. The protocol allows a program running on one computer to call a subroutine on another without the programmer having to program for this interaction.

It is worth noting that two or more applications are distributed on different servers, and the calls between them behave like local method calls.

There are many RPC frameworks, such as Ali’s Dubbo, Google’s gRPC, Go language RPCX, Apache Thrift. There’s Spring Cloud, of course, but for Spring Cloud, RPC is just one of its functional modules.

Complexity aside, what does it take to implement a basic, simple RPC?

  • A dynamic proxy
  • reflection
  • Serialization, deserialization
  • Network communication
  • codec
  • Service discovery and registration
  • Heartbeat and link detection
  • .

Let’s take a look at the code that strings these technical points together to implement our own RPC.

Two, environmental preparation

Before I begin, I’ll describe the software environment used.

SpringBoot, Netty, ZooKeeper, ZKClient, fastJSON

  • SpringBoot project foundation framework, convenient into JAR packages, easy to test.
  • Netty communication server
  • Discover and register the ZooKeeper service
  • Zkclient ZooKeeper client
  • Fastjson serialization, deserialization

RPC producer

1. Service interface API

Throughout RPC, we divide into producers and consumers. First, they share a common service interface API. In this case, we have a Service interface that operates on user information.

public interface InfoUserService {
    List<InfoUser> insertInfoUser(InfoUser infoUser);
    InfoUser getInfoUserById(String id);
    void deleteInfoUserById(String id);
    String getNameById(String id);
    Map<String,InfoUser> getAllUser();
}
Copy the code

2. Service class implementation

As a producer, it certainly needs to have an implementation class, so we create the InfoUserServiceImpl implementation class, annotate it as a service for RPC, and then register it with Spring’s Bean container. In this case, we use infoUserMap as a database to store user information.

package com.viewscenes.netsupervisor.service.impl; @RpcService public class InfoUserServiceImpl implements InfoUserService { Logger logger = LoggerFactory.getLogger(this.getClass()); Map<String,InfoUser> infoUserMap = new HashMap<>(); public List<InfoUser> insertInfoUser(InfoUser infoUser) { logger.info("Add user info :{}", JSONObject.toJSONString(infoUser));
        infoUserMap.put(infoUser.getId(),infoUser);
        return getInfoUserList();
    }
    public InfoUser getInfoUserById(String id) {
        InfoUser infoUser = infoUserMap.get(id);
        logger.info("Query user ID:{}",id);
        return infoUser;
    }

    public List<InfoUser> getInfoUserList() {
        List<InfoUser> userList = new ArrayList<>();
        Iterator<Map.Entry<String, InfoUser>> iterator = infoUserMap.entrySet().iterator();
        while (iterator.hasNext()){
            Map.Entry<String, InfoUser> next = iterator.next();
            userList.add(next.getValue());
        }
        logger.info("Return number of user information records :{}",userList.size());
        return userList;
    }
    public void deleteInfoUserById(String id) {
        logger.info("Delete user information :{}",JSONObject.toJSONString(infoUserMap.remove(id)));
    }
    public String getNameById(String id){
        logger.info("Query user name by ID :{}",id);
        return infoUserMap.get(id).getName();
    }
    public Map<String,InfoUser> getAllUser(){
        logger.info("Query all user information {}",infoUserMap.keySet().size());
        returninfoUserMap; }}Copy the code

A meta-annotation is defined as follows:

package com.viewscenes.netsupervisor.annotation;

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Component
public @interface RpcService {}
Copy the code

Request information and return information

All the request information and return information are represented by two Javabeans. The important thing is that the returned information has the ID of the requested information.

package com.viewscenes.netsupervisor.entity; public class Request { private String id; private String className; // Class name private String methodName; Private Class<? >[] parameterTypes; // Private Object[] parameters; // Parameter list get/set. }Copy the code

package com.viewscenes.netsupervisor.entity;
public class Response {
	private String requestId;
	private int code;
	private String error_msg;
	private Object data;
	get/set. }Copy the code

4. Netty server

As a high performance NIO communication framework, Netty has appeared in many RPC frameworks. We also use it as a communication server. Zookeeper’s registered address and Netty’s communication server address are two important configuration files.

TOMCAT server port. The port = 8001# Registered address of ZooKeeperRegistry. Address = 192.168.245.131:2181192168. 245.131:2182192168 245.131:2183RPC service provider addressRPC. Server. The address = 192.168.197.1:18868Copy the code

For ease of administration, we register it as a Bean and implement the ApplicationContextAware interface, fishing out the service class annotated by @rpcService and caching it for consumer invocation. At the same time, the server also checks the heartbeat of the client. If no data is read or written for more than 60 seconds, the connection is closed.

package com.viewscenes.netsupervisor.netty.server;
@Component
public class NettyServer implements ApplicationContextAware,InitializingBean{

    private static final Logger logger = LoggerFactory.getLogger(NettyServer.class);
    private static final EventLoopGroup bossGroup = new NioEventLoopGroup(1);
    private static final EventLoopGroup workerGroup = new NioEventLoopGroup(4);

    private Map<String, Object> serviceMap = new HashMap<>();

    @Value("${rpc.server.address}")
    private String serverAddress;

    @Autowired
    ServiceRegistry registry;

    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        Map<String, Object> beans = applicationContext.getBeansWithAnnotation(RpcService.class);
        for(Object serviceBean:beans.values()){ Class<? > clazz = serviceBean.getClass(); Class<? >[] interfaces = clazz.getInterfaces();for(Class<? > inter : interfaces){ String interfaceName = inter.getName(); logger.info("Load service class: {}", interfaceName);
                serviceMap.put(interfaceName, serviceBean);
            }
        }
        logger.info("All service interfaces loaded :{}", serviceMap);
    }
    public void afterPropertiesSet() throws Exception {
        start();
    }
    public void start(){
        final NettyServerHandler handler = new NettyServerHandler(serviceMap);
        new Thread(() -> {
            try {
                ServerBootstrap bootstrap = new ServerBootstrap();
                bootstrap.group(bossGroup,workerGroup).
                        channel(NioServerSocketChannel.class).
                        option(ChannelOption.SO_BACKLOG,1024).
                        childOption(ChannelOption.SO_KEEPALIVE,true).
                        childOption(ChannelOption.TCP_NODELAY,trueChildHandler (new ChannelInitializer<SocketChannel>() {// After creating a NIOSocketChannel, initialize the NIOSocketChannel. Set its ChannelHandler to the ChannelPipeline, Protected void initChannel(SocketChannel) throws Exception {ChannelPipeline pipeline = channel.pipeline(); pipeline.addLast(new IdleStateHandler(0, 0, 60)); pipeline.addLast(new JSONEncoder()); pipeline.addLast(new JSONDecoder()); pipeline.addLast(handler); }}); String[] array = serverAddress.split(":");
                String host = array[0];
                int port = Integer.parseInt(array[1]);
                ChannelFuture cf = bootstrap.bind(host,port).sync();
                logger.info("RPC server started. Listening port :"+port); registry.register(serverAddress); Cf.channel ().closeFuture().sync(); // Wait for the server listening port to close cf.channel().closeFuture().sync(); } catch (Exception e) { e.printStackTrace(); bossGroup.shutdownGracefully(); workerGroup.shutdownGracefully(); } }).start(); }}Copy the code

The above code starts the Netty server. In the constructor of the processor, we first pass in the Map of the service Bean. All processing must be based on this Map to find the corresponding implementation class. In channelRead, get the information for the request method, and then get the return value by calling the reflection method.

package com.viewscenes.netsupervisor.netty.server;
@ChannelHandler.Sharable
public class NettyServerHandler extends ChannelInboundHandlerAdapter {

    private final Logger logger = LoggerFactory.getLogger(NettyServerHandler.class);
    private final Map<String, Object> serviceMap;

    public NettyServerHandler(Map<String, Object> serviceMap) {
        this.serviceMap = serviceMap;
    }
    public void channelActive(ChannelHandlerContext ctx)   {
        logger.info("Client connection successful!"+ctx.channel().remoteAddress());
    }
    public void channelInactive(ChannelHandlerContext ctx)   {
        logger.info("Client disconnected! {}",ctx.channel().remoteAddress());
        ctx.channel().close();
    }
    public void channelRead(ChannelHandlerContext ctx, Object msg)   {
        Request request = JSON.parseObject(msg.toString(),Request.class);

        if ("heartBeat".equals(request.getMethodName())) {
            logger.info("Client heartbeat information..."+ctx.channel().remoteAddress());
        }else{
            logger.info(RPC client request interface:+request.getClassName()+"Method name :"+request.getMethodName());
            Response response = new Response();
            response.setRequestId(request.getId());
            try {
                Object result = this.handler(request);
                response.setData(result);
            } catch (Throwable e) {
                e.printStackTrace();
                response.setCode(1);
                response.setError_msg(e.toString());
                logger.error("RPC Server handle request error",e); } ctx.writeAndFlush(response); }} /** * By reflection, execute the local method * @param request * @return
     * @throws Throwable
     */
    private Object handler(Request request) throws Throwable{
        String className = request.getClassName();
        Object serviceBean = serviceMap.get(className);

        if(serviceBean! =null){ Class<? > serviceClass = serviceBean.getClass(); String methodName = request.getMethodName(); Class<? >[] parameterTypes = request.getParameterTypes(); Object[] parameters = request.getParameters(); Method method = serviceClass.getMethod(methodName, parameterTypes); method.setAccessible(true);
            return method.invoke(serviceBean, getParameters(parameterTypes,parameters));
        }else{
            throw new Exception("Service interface not found, please check configuration! :"+className+"#"+request.getMethodName()); }} /** * get the parameter list * @param parameterTypes * @param parameters * @return*/ private Object[] getParameters(Class<? >[] parameterTypes,Object[] parameters){if (parameters==null || parameters.length==0){
            return parameters;
        }else{
            Object[] new_parameters = new Object[parameters.length];
            for(int i=0; i<parameters.length; i++){ new_parameters[i] = JSON.parseObject(parameters[i].toString(),parameterTypes[i]); }return new_parameters;
        }
    }
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt)throws Exception {
        if (evt instanceof IdleStateEvent){
            IdleStateEvent event = (IdleStateEvent)evt;
            if (event.state()== IdleState.ALL_IDLE){
                logger.info("The client has not read or written data for more than 60 seconds. Close the connection.,ctx.channel().remoteAddress()); ctx.channel().close(); }}else{ super.userEventTriggered(ctx,evt); } } public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { logger.info(cause.getMessage()); ctx.close(); }}Copy the code

4. Service registration

We started the Netty communication server and loaded the service implementation class into the cache to be called upon request. In this step, we register the service. To simplify the process, we only register the listening address of the communication server. In the code above, after bind we execute registry. Register (serverAddress); Zookeeper registers IP ports listened by Netty with ZooKeeper.

package com.viewscenes.netsupervisor.registry;
@Component
public class ServiceRegistry {
    Logger logger = LoggerFactory.getLogger(this.getClass());
    @Value("${registry.address}")
    private String registryAddress;
    private static final String ZK_REGISTRY_PATH = "/rpc";

    public void register(String data) {
        if(data ! = null) { ZkClient client = connectServer();if(client ! = null) { AddRootNode(client); createNode(client, data); }}} // Connect to ZooKeeper private ZkClientconnectServer() {ZkClient client = new ZkClient (registryAddress, 20000200, 00);returnclient; } // Create a root directory/RPC private void AddRootNode(ZkClient client){Boolean exists = client.exists(ZK_REGISTRY_PATH);if(! exists){ client.createPersistent(ZK_REGISTRY_PATH); logger.info("Create primary ZooKeeper node {}",ZK_REGISTRY_PATH); }} // In the/RPC root directory, Create (ZK_REGISTRY_PATH +) private void createNode(ZkClient client, String data) {String path = client.create(ZK_REGISTRY_PATH +"/provider", data, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
        logger.info("Create zooKeeper data Node ({} => {})", path, data); }}Copy the code

It is important to note that the child nodes must be temporary. In this way, when the producer side is down, the consumer can be notified to remove the service from the service list. At this point, the producer side is complete. Let’s take a look at its startup log:

Load the service class: com.viewscenes.netsupervisor.service.InfoUserService Have to load all the service interface: {com.viewscenes.netsupervisor.service.InfoUserService=com.viewscenes.net supervisor. The service. The impl. InfoUserServic eImpl@46cc127b} Initializing ExecutorService'applicationTaskExecutor'
Tomcat started on port(s): 8001 (http) with context path ' '
Started RpcProviderApplication in2.003 seconds (JVM is runningfor3.1) RPC server starts. Listen on port: 18868 Starting ZkClient event thread. The Socket connection established to 192.168.245.131/192.168.245.131:2183, Initiating the session session establishment complete on server 192.168.245.131/192.168.245.131:2183, sessionid = 0x367835b48970010, Negotiated Timeout = 4000 ZooKeeper state changed (SyncConnected) Create zooKeeper master node/RPC create ZooKeeper data node (/ RPC/provider0000000000 = > 192.168.197.1:28868).Copy the code

4. RPC consumers

First, we need to add the producer-side service interface API, known as InfoUserService. Put it in the same directory on the consumer side. Different path, call will not find oh.

1, agents,

One of the goals of RPC is that “no programmer needs to program for this interaction.” So, we call it as if we were calling a local method. Like this:

@Controller
public class IndexController {	
	@Autowired
    InfoUserService userService;
	
	@RequestMapping("getById")
    @ResponseBody
    public InfoUser getById(String id){
        logger.info("Query user information by ID :{}",id);
        returnuserService.getInfoUserById(id); }}Copy the code

So, here’s the problem. There is no implementation of this interface on the consumer side, how to call it? Here, the first is the agent. I use here is the Spring factory Bean mechanism to create the proxy object, involving more code, is not reflected in the article, if you do not understand the students, please imagine, MyBatis Mapper interface how to be called. Mybatis source code analysis (four) Mapper interface method is how to be called

In summary, when the userService method is called, the invoke method of the proxy object is called. Here, the request information is encapsulated and Netty’s client methods are called to send the message. It then returns the corresponding object based on the method return value type.

package com.viewscenes.netsupervisor.configurer.rpc; @Component public class RpcFactory<T> implements InvocationHandler { @Autowired NettyClient client; Logger logger = LoggerFactory.getLogger(this.getClass()); public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { Request request = new Request(); request.setClassName(method.getDeclaringClass().getName()); request.setMethodName(method.getName()); request.setParameters(args); request.setParameterTypes(method.getParameterTypes()); request.setId(IdUtil.getId()); Object result = client.send(request); Class<? >returnType = method.getReturnType();

        Response response = JSON.parseObject(result.toString(), Response.class);
        if (response.getCode()==1){
            throw new Exception(response.getError_msg());
        }
        if (returnType.isPrimitive() || String.class.isAssignableFrom(returnType)){
            return response.getData();
        }else if (Collection.class.isAssignableFrom(returnType)){
            return JSONArray.parseArray(response.getData().toString(),Object.class);
        }else if(Map.class.isAssignableFrom(returnType)){
            return JSON.parseObject(response.getData().toString(),Map.class);
        }else{
            Object data = response.getData();
            return JSONObject.parseObject(data.toString(), returnType); }}}Copy the code

2. Service discovery

On the producer side, we registered the service IP ports with ZooKeeper, so in this case, we’re going to get the service address and connect through Netty. It is important to also listen for child node changes in the root directory so that the consumer side is aware as producers go online and offline.

package com.viewscenes.netsupervisor.connection;

@Component
public class ServiceDiscovery {

    @Value("${registry.address}") private String registryAddress; @Autowired ConnectManage connectManage; Private volatile List<String> addressList = new ArrayList<>(); private static final String ZK_REGISTRY_PATH ="/rpc";
    private ZkClient client;

    Logger logger = LoggerFactory.getLogger(this.getClass());

    @PostConstruct
    public void init(){
        client = connectServer();
        if(client ! = null) { watchNode(client); }} // Connect to ZooKeeper private ZkClientconnectServer() {ZkClient client = new ZkClient (registryAddress, 30000300, 00);returnclient; Private void watchNode(final ZkClient client) {List<String> nodeList = client.subscribeChildChanges(ZK_REGISTRY_PATH, (s, nodes) -> { logger.info("Monitoring child node data changes {}",JSONObject.toJSONString(nodes));
            addressList.clear();
            getNodeData(nodes);
            updateConnectedServer();
        });
        getNodeData(nodeList);
        logger.info("List of discovered services... {}", JSONObject.toJSONString(addressList)); updateConnectedServer(); } // Connect to producer side service private voidupdateConnectedServer(){
        connectManage.updateConnectServer(addressList);
    }

    private void getNodeData(List<String> nodes){
        logger.info("/ RPC child node data is :{}", JSONObject.toJSONString(nodes));
        for(String node:nodes){
            String address = client.readData(ZK_REGISTRY_PATH+"/"+node); addressList.add(address); }}}Copy the code

Among them, the connectManage. UpdateConnectServer (expressions such as addressList); To connect to the Netty service on the producer end according to the service ADDRESS. Then create a list of channels, and select one of them to communicate with the producer side when sending a message.

Netty client

The Netty client has two important methods. One is to connect to the server according to the IP port, return Channel, and join the connection manager. One is to send request data using a Channel. At the same time, as the client, when idle to send heartbeat information to the server.

package com.viewscenes.netsupervisor.netty.client;

@Component
public class NettyClient {
    Logger logger = LoggerFactory.getLogger(this.getClass());
    private EventLoopGroup group = new NioEventLoopGroup(1);
    private Bootstrap bootstrap = new Bootstrap();
    @Autowired
    NettyClientHandler clientHandler;
    @Autowired
    ConnectManage connectManage;
   
    public Object send(Request request) throws InterruptedException{

        Channel channel = connectManage.chooseChannel();
        if(channel! =null && channel.isActive()) { SynchronousQueue<Object> queue = clientHandler.sendRequest(request,channel); Object result = queue.take();return JSONArray.toJSONString(result);
        }else{
            Response res = new Response();
            res.setCode(1);
            res.setError_msg("Incorrect connection to server. Please check the configuration information!");
            return JSONArray.toJSONString(res);
        }
    }
    public Channel doConnect(SocketAddress address) throws InterruptedException {
        ChannelFuture future = bootstrap.connect(address);
        Channel channel = future.sync().channel();
        returnchannel; }... Other methods omitted}Copy the code

We must focus on the Send method, which is called from the invoke method of the proxy object. You first poll to select a Channel from the connector, and then send the data. However, since Netty is an asynchronous operation, we have to go synchronous, meaning we wait for the producer to return data before proceeding. Here we use the SynchronousQueue, whose take method blocks until there is something to read. Then in the handler, it takes the return message and writes it to the queue, and the take method returns.

package com.viewscenes.netsupervisor.netty.client;
@Component
@ChannelHandler.Sharable
public class NettyClientHandler extends ChannelInboundHandlerAdapter {

    @Autowired
    NettyClient client;
    @Autowired
    ConnectManage connectManage;
    Logger logger = LoggerFactory.getLogger(this.getClass());
    private ConcurrentHashMap<String,SynchronousQueue<Object>> queueMap = new ConcurrentHashMap<>();

    public void channelActive(ChannelHandlerContext ctx)   {
        logger.info("RPC server connected.{}",ctx.channel().remoteAddress());
    }
    public void channelInactive(ChannelHandlerContext ctx)   {
        InetSocketAddress address =(InetSocketAddress) ctx.channel().remoteAddress();
        logger.info("Disconnected from the RPC server."+address);
        ctx.channel().close();
        connectManage.removeChannel(ctx.channel());
    }
    public void channelRead(ChannelHandlerContext ctx, Object msg)throws Exception {
        Response response = JSON.parseObject(msg.toString(),Response.class);
        String requestId = response.getRequestId();
        SynchronousQueue<Object> queue = queueMap.get(requestId);
        queue.put(response);
        queueMap.remove(requestId);
    }
    public SynchronousQueue<Object> sendRequest(Request request,Channel channel) {
        SynchronousQueue<Object> queue = new SynchronousQueue<>();
        queueMap.put(request.getId(), queue);
        channel.writeAndFlush(request);
        return queue;
    }
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt)throws Exception {
        logger.info("No read or write to RPC server for more than 30 seconds! Will send a heartbeat message...");
        if (evt instanceof IdleStateEvent){
            IdleStateEvent event = (IdleStateEvent)evt;
            if (event.state()== IdleState.ALL_IDLE){
                Request request = new Request();
                request.setMethodName("heartBeat"); ctx.channel().writeAndFlush(request); }}else{
            super.userEventTriggered(ctx,evt);
        }
    }
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause){
        logger.info("RPC communication server is abnormal.{}",cause); ctx.channel().close(); }}Copy the code

At this point, the consumer side is also basically complete. Again, let’s look at the startup log:

Waiting forKeeper state SyncConnected Opening a socket connection to the server 192.168.139.129/192.168.139.129:2181. Will not attempt to Authenticate using SASL (unknown error) Socket connection established to 192.168.139.129/192.168.139.129:2181, Initiating the session session establishment complete on server 192.168.139.129/192.168.139.129:2181, Sessionid = 0x100000273ba002C, Negotiated Timeout = 20000 ZooKeeper state changed (SyncConnected)/Child node data is :["provider0000000015"] Discovered service list... ["192.168.100.74:18868"] Joins a Channel to the connection manager./192.168.100.74:18868 Connected to the RPC server./192.168.100.74:18868 Initializing ExecutorService'applicationTaskExecutor'
Tomcat started on port(s): 7002 (http) with context path ' '
Started RpcConsumerApplication in4.218 seconds (JVM is runningfor 5.569)
Copy the code

Five, the test

Let’s take the two methods in Controller as an example. We start 100 threads calling insertInfoUser, and then start 1000 threads calling getAllUser.

public class IndexController {

    Logger logger = LoggerFactory.getLogger(this.getClass());
    @Autowired
    InfoUserService userService;

    @RequestMapping("insert")
    @ResponseBody
    public List<InfoUser> getUserList() throws InterruptedException {
        long start = System.currentTimeMillis();
        int thread_count = 100;
        CountDownLatch countDownLatch = new CountDownLatch(thread_count);
        for(int i=0; i<thread_count; i++){ new Thread(() -> { InfoUser infoUser = new InfoUser(IdUtil.getId(),"Jeen"."BeiJing");
                List<InfoUser> users = userService.insertInfoUser(infoUser);
                logger.info(Return user info record :{}, JSON.toJSONString(users));
                countDownLatch.countDown();
            }).start();
        }
        countDownLatch.await();
        long end = System.currentTimeMillis();
        logger.info("Thread count :{}, execution time :{}",thread_count,(end-start));
        return null;
    }
	@RequestMapping("getAllUser")
    @ResponseBody
    public Map<String,InfoUser> getAllUser() throws InterruptedException {

        long start = System.currentTimeMillis();
        int thread_count = 1000;
        CountDownLatch countDownLatch = new CountDownLatch(thread_count);
        for(int i=0; i<thread_count; i++){ new Thread(() -> { Map<String, InfoUser> allUser = userService.getAllUser(); logger.info("Query all user information: {}",JSONObject.toJSONString(allUser));
                countDownLatch.countDown();
            }).start();
        }
        countDownLatch.await();
        long end = System.currentTimeMillis();
        logger.info("Thread count :{}, execution time :{}",thread_count,(end-start));

        returnnull; }}Copy the code

The results are as follows:

Six, summarized

This article briefly introduces the whole process of RPC. If you are learning about RPC, you can follow the examples in this article to implement it yourself. After writing this, you will have a better understanding of RPC.

Producer-side process:

  • Load the service and cache it
  • Start communication Server (Netty)
  • Service registration (add the mailing address to ZooKeeper and also add the loaded service to zooKeeper)
  • Reflection, local call

Consumer-side process:

  • Proxy service interface
  • Service discovery (Connect to ZooKeeper and get the service address list)
  • Remote invocation (polling the list of producer services, sending messages)

Limited to space, in this paper, the code is not complete, if necessary, visit: https://github.com/taoxun/simple_rpc, or add the author WeChat number: public blog > < beautiful place), access to complete the project.