1, the principle of

The dubbo architecture is as follows:

By reading the Dubbo source code, all RPC method calls are intercepted by MonitorFilter,

MonitorFilter.invoke()

public Result invoke(Invoker<? > invoker, Invocation invocation) throws RpcException {if (invoker.getUrl().hasParameter("monitor")) {
            RpcContext context = RpcContext.getContext();
            long start = System.currentTimeMillis();
            this.getConcurrent(invoker, invocation).incrementAndGet();

            Result var7;
            try {
                Result result = invoker.invoke(invocation);
            this.collect(invoker, invocation, result, context, start, false);
                var7 = result;
            } catch (RpcException var11) {
                this.collect(invoker, invocation, (Result)null, context, start, true);
                throw var11;
            } finally {
                this.getConcurrent(invoker, invocation).decrementAndGet();
            }

            return var7;
        } else {
            returninvoker.invoke(invocation); }}Copy the code

For services configured with monitoring, basic statistics for some methods are collected.

MonitorFilter.collect()

private void collect(Invoker<? > invoker, Invocation invocation, Result result, RpcContext context, long start, boolean error) { try { long elapsed = System.currentTimeMillis() - start; int concurrent = this.getConcurrent(invoker, invocation).get(); String application = invoker.getUrl().getParameter("application");
            String service = invoker.getInterface().getName();
            String method = RpcUtils.getMethodName(invocation);
            URL url = invoker.getUrl().getUrlParameter("monitor");
            Monitor monitor = this.monitorFactory.getMonitor(url);
            int localPort;
            String remoteKey;
            String remoteValue;
            if ("consumer".equals(invoker.getUrl().getParameter("side"))) {
                context = RpcContext.getContext();
                localPort = 0;
                remoteKey = "provider";
                remoteValue = invoker.getUrl().getAddress();
            } else {
                localPort = invoker.getUrl().getPort();
                remoteKey = "consumer";
                remoteValue = context.getRemoteHost();
            }

            String input = "";
            String output = "";
            if (invocation.getAttachment("input") != null) {
                input = invocation.getAttachment("input");
            }

            if(result ! = null && result.getAttachment("output") != null) {
                output = result.getAttachment("output");
            }

            monitor.collect(new URL("count", NetUtils.getLocalHost(), localPort, service + "/" + method, new String[]{"application", application, "interface", service, "method", method, remoteKey, remoteValue, error ? "failure" : "success"."1"."elapsed", String.valueOf(elapsed), "concurrent", String.valueOf(concurrent), "input", input, "output", output}));
        } catch (Throwable var21) {
            logger.error("Failed to monitor count service " + invoker.getUrl() + ", cause: "+ var21.getMessage(), var21); }}Copy the code

DubboMonitor collects simple statistics on the collected data, such as the number of successes, failures, and call time. After collecting statistics, the data is stored locally.

DubboMonitor.collect()

public void collect(URL url) {
        int success = url.getParameter("success", 0);
        int failure = url.getParameter("failure", 0);
        int input = url.getParameter("input", 0);
        int output = url.getParameter("output", 0);
        int elapsed = url.getParameter("elapsed", 0);
        int concurrent = url.getParameter("concurrent", 0);
        Statistics statistics = new Statistics(url);
        AtomicReference<long[]> reference = (AtomicReference)this.statisticsMap.get(statistics);
        if (reference == null) {
            this.statisticsMap.putIfAbsent(statistics, new AtomicReference());
            reference = (AtomicReference)this.statisticsMap.get(statistics);
        }

        long[] update = new long[10];

        long[] current;
        do {
            current = (long[])reference.get();
            if (current == null) {
                update[0] = (long)success;
                update[1] = (long)failure;
                update[2] = (long)input;
                update[3] = (long)output;
                update[4] = (long)elapsed;
                update[5] = (long)concurrent;
                update[6] = (long)input;
                update[7] = (long)output;
                update[8] = (long)elapsed;
                update[9] = (long)concurrent;
            } else{ update[0] = current[0] + (long)success; update[1] = current[1] + (long)failure; update[2] = current[2] + (long)input; update[3] = current[3] + (long)output; update[4] = current[4] + (long)elapsed; update[5] = (current[5] + (long)concurrent) / 2L; update[6] = current[6] > (long)input ? current[6] : (long)input; update[7] = current[7] > (long)output ? current[7] : (long)output; update[8] = current[8] > (long)elapsed ? current[8] : (long)elapsed; update[9] = current[9] > (long)concurrent ? current[9] : (long)concurrent; }}while(! reference.compareAndSet(current, update)); }Copy the code

DubboMonitor has asynchronous threads that periodically (every minute by default) send collected data to the remote monitoring service.

 public DubboMonitor(Invoker<MonitorService> monitorInvoker, MonitorService monitorService) {
        this.monitorInvoker = monitorInvoker;
        this.monitorService = monitorService;
        this.monitorInterval = (long)monitorInvoker.getUrl().getPositiveParameter("interval", 60000);
        this.sendFuture = this.scheduledExecutorService.scheduleWithFixedDelay(new Runnable() {
            public void run() {
                try {
                    DubboMonitor.this.send();
                } catch (Throwable var2) {
                    DubboMonitor.logger.error("Unexpected error occur at send statistic, cause: " + var2.getMessage(), var2);
                }

            }
        }, this.monitorInterval, this.monitorInterval, TimeUnit.MILLISECONDS);
    }
Copy the code

Call the remote MonitorService.collect method and set the locally cached data to zero.

DubboMonitor.send()

 public void send() {
        if (logger.isInfoEnabled()) {
            logger.info("Send statistics to monitor " + this.getUrl());
        }

        String timestamp = String.valueOf(System.currentTimeMillis());
        Iterator i$ = this.statisticsMap.entrySet().iterator();

        while(i$.hasNext()) {
            Entry<Statistics, AtomicReference<long[]>> entry = (Entry)i$.next();
            Statistics statistics = (Statistics)entry.getKey();
            AtomicReference<long[]> reference = (AtomicReference)entry.getValue();
            long[] numbers = (long[])reference.get();
            long success = numbers[0];
            long failure = numbers[1];
            long input = numbers[2];
            long output = numbers[3];
            long elapsed = numbers[4];
            long concurrent = numbers[5];
            long maxInput = numbers[6];
            long maxOutput = numbers[7];
            long maxElapsed = numbers[8];
            long maxConcurrent = numbers[9];
            URL url = statistics.getUrl().addParameters(new String[]{"timestamp", timestamp, "success", String.valueOf(success), "failure", String.valueOf(failure), "input", String.valueOf(input), "output", String.valueOf(output), "elapsed", String.valueOf(elapsed), "concurrent", String.valueOf(concurrent), "max.input", String.valueOf(maxInput), "max.output", String.valueOf(maxOutput), "max.elapsed", String.valueOf(maxElapsed), "max.concurrent", String.valueOf(maxConcurrent)});
            this.monitorService.collect(url);
            long[] update = new long[10];

            while(true) {
                long[] current = (long[])reference.get();
                if (current == null) {
                    update[0] = 0L;
                    update[1] = 0L;
                    update[2] = 0L;
                    update[3] = 0L;
                    update[4] = 0L;
                    update[5] = 0L;
                } else {
                    update[0] = current[0] - success;
                    update[1] = current[1] - failure;
                    update[2] = current[2] - input;
                    update[3] = current[3] - output;
                    update[4] = current[4] - elapsed;
                    update[5] = current[5] - concurrent;
                }

                if (reference.compareAndSet(current, update)) {
                    break; }}}}Copy the code

Dubbo monitors the main open source projects, all implement MonitorService interface for monitoring, the differences are data storage, report statistical logic, the basic principles are similar.

public interface MonitorService {
    String APPLICATION = "application";
    String INTERFACE = "interface";
    String METHOD = "method";
    String GROUP = "group";
    String VERSION = "version";
    String CONSUMER = "consumer";
    String PROVIDER = "provider";
    String TIMESTAMP = "timestamp";
    String SUCCESS = "success";
    String FAILURE = "failure";
    String INPUT = "input";
    String OUTPUT = "output";
    String ELAPSED = "elapsed";
    String CONCURRENT = "concurrent";
    String MAX_INPUT = "max.input";
    String MAX_OUTPUT = "max.output";
    String MAX_ELAPSED = "max.elapsed";
    String MAX_CONCURRENT = "max.concurrent";

    void collect(URL var1);

    List<URL> lookup(URL var1);
}
Copy the code

2. Monitor the selection

Mainstream DuBBo monitoring mainly includes:

• dubbo-monitor
• dubbo-d-monitor
• dubbokeeper
• dubbo-monitor-simple

Here’s a simple comparison:

Dubbokeeper >dubbo-monitor> dubbod-monitor. Therefore, DubboKeeper is selected as the dubbo service monitoring solution.

3, deployment,

We use mongodb storage solution and single-node deployment.

Environment: JDK1.8 or higher (low version not tested), install Tomcat, install ZooKeeper and start, install mongodb

1, get the source github.com/dubboclub/d…

Dubbokeeper-master = dubbokeeper-master = dubbokeeper-master = dubbo-mongodb.properties = dubbokeeper-master = dubbokeeper-master = dubbokeeper-master = dubbo-mongodb.properties = dubbokeeper-master = dubbokeeper-master = dubbokeeper-master = dubbo-mongodb.properties

Run the \dubbokeeper-master\install-mongodb. Sh command to generate a target directory, which contains the following three folders and a compressed package

 archive-tmp
 mongodb-dubbokeeper-server
 mongodb-dubbokeeper-ui
 mongodb-dubbokeeper-server.tar.gz
Copy the code

3. Run the mongodb-dubbokeeper-server/bin/start-mongodb.

4. Copy the war package in mongodb- dubbokeeper-UI to the Tomcat webapps directory and start Tomcat.

5, and finally, open a browser, type http://localhost:8080/dubbokeeper-ui-1.0.1.

In the business code, you only need to configure the Dubbo monitoring connection to the registry to complete the monitoring data collection.

<dubbo:monitor protocol="registry"/>
Copy the code

Main configuration information:

dubbo.application.name=mongodb-monitor
dubbo.application.owner=bieber
dubbo.registry.address=zookeeper://*.*.*.*:2181?backup=*.*.*.*:2181,*.*.*.*:2181
dubbo.protocol.name=dubbo
dubbo.protocol.port=20884
dubbo.protocol.dubbo.payload=20971520

Dubbo Data collection period in milliseconds
monitor.collect.interval=60000

#use netty4
dubbo.provider.transporter=netty4

Dubbokeeper writes mongodb cycles in seconds
monitor.write.interval=60

# mongdb configuration
dubbo.monitor.mongodb.url=localhost
dubbo.monitor.mongodb.port=27017
dubbo.monitor.mongodb.dbname=dubbokeeper
dubbo.monitor.mongodb.username=
dubbo.monitor.mongodb.password=
dubbo.monitor.mongodb.storage.timeout=60000
Copy the code

4. Main functions

On the home page, you can see the overall application information (distinguishing application providers and consumers), the number of services, node deployment information and dependency diagram, etc.

Admin provides most of the functionality of all native Dubo-Admin.

ZooPeeper displays zooKeeper node information

Monitor displays dubbo monitoring information

Application overview information can be filtered by time:

Detailed application information, including interface time, concurrency, failure, and success:

Method level Overview and details:

5, encountered pit

The default value of monitor.write. Interval is set to 6000. The unit of source code discovery is seconds.

2. Dubbokeeper does not index Collections by default. It will be very slow to open collections if there is a large amount of data.

import pymongo
from pymongo import MongoClient
import time
import datetime
import sys
import os


client = MongoClient('127.0.0.1', 27017)
db = client['dubbokeeper']

collectionlist = db.collection_names()

for collection in collectionlist:
    ifcollection! ='application':
        db[collection].ensure_index([("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING)])
        db[collection].ensure_index([("method",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING),("method",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("concurrent",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("elapsed",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("failureCount",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("successCount",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING),("elapsed",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING),("concurrent",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING),("failureCount",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        db[collection].ensure_index([("serviceInterface",pymongo.DESCENDING),("successCount",pymongo.DESCENDING),("timestamp",pymongo.DESCENDING)])
        
print 'success'
Copy the code

3. Generally, historical data need not be saved for a long time. At present, we keep online data for 2 weeks and provide the following script to delete data regularly.

import pymongo
from pymongo import MongoClient
import time
import datetime
import sys
import os

day=int(sys.argv[1])
print day
timestamp = time.time()*1000-1000*24*3600*day

print timestamp

client = MongoClient('127.0.0.1', 27017)
db = client['dubbokeeper']

collectionlist = db.collection_names()

for collection in collectionlist:
    ifcollection! ='application':
        db[collection].remove({"timestamp": {"$lt": timestamp}})

print 'clean mongodb data success'
Copy the code

Clear 15 days’ data periodically every day

0 3 * * * python /home/monitor/shell/clean-mongodb.py 15
Copy the code

4, mongodb cache eats memory, it is best to configure more than 8GB server, or a large amount of cluster deployment can be considered

5. Dubbokeeper-ui native interaction is a bit clunky, with some pages iterating through all application data, which is inefficient. If there are too many applications, they may not be opened due to timeout. The server team has simply optimized the interaction and can only view one application or one interface at a time. If you have any requirements, you can leave a message, and we will open source it later.