`OAL`Basic knowledge of

Basic introduction

OAL(Observability Analysis Language) is a Language for analyzing streaming data.

Because OAL focuses on metrics that measure services, Service Instances, and endpoints, it is simple to learn and use.

OAL converts OAL scripts into dynamically generated class files based on AlTLr and Javassist.

Since version 6.3, the OAL engine is built into the OAP server as oal-RT (OAL Runtime). OAL script location OAL configuration directory (/config/ OAL), the user can change the script and restart it. Note: OAL scripting is still a compiled language, and Oal-RT dynamically generates Java code.

If you set the environment variable SW_OAL_ENGINE_DEBUG=Y, you can find the generated Class file in the oal-rt directory in your working directory.

grammar

// Declare an indicatorMETRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...] ))// Retrieve data from a SCOPE
[.filter(FIELD OP [INT | STRING])] // Some data can be filtered out.FUNCTION([PARAM][, PARAM ...] )// Aggregate the data using an aggregate function

// Disable an indicator
disable(METRICS_NAME);
Copy the code

The grammar of case

oap-server/server-bootstrap/src/main/resources/oal/java-agent.oal

// Get data from used in ServiceInstanceJVMMemory, only need data with heapStatus true and take the average value of long
instance_jvm_memory_heap = from(ServiceInstanceJVMMemory.used).filter(heapStatus == true).longAvg();
Copy the code

org.apache.skywalking.oap.server.core.source.ServiceInstanceJVMMemory

@ScopeDeclaration(id = SERVICE_INSTANCE_JVM_MEMORY, name = "ServiceInstanceJVMMemory", catalog = SERVICE_INSTANCE_CATALOG_NAME)
@ScopeDefaultColumn.VirtualColumnDefinition(fieldName = "entityId", columnName = "entity_id", isID = true, type = String.class)
public class ServiceInstanceJVMMemory extends Source {
    @Override
    public int scope(a) {
        return DefaultScopeDefine.SERVICE_INSTANCE_JVM_MEMORY;
    }

    @Override
    public String getEntityId(a) {
        return String.valueOf(id);
    }

    @Getter @Setter
    private String id;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "name", requireDynamicActive = true)
    private String name;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_name", requireDynamicActive = true)
    private String serviceName;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_id")
    private String serviceId;
    @Getter @Setter
    private boolean heapStatus;
    @Getter @Setter
    private long init;
    @Getter @Setter
    private long max;
    @Getter @Setter
    private long used;
    @Getter @Setter
    private long committed;
}
Copy the code

Official documentation for reference: Observability Analysis Language

Start with a case study`OAL`The principle of

Missing monitoring of class loading information

The default APM/Instance page lacks information about the JVM Class (as shown in the figure below), so this time it will be filled in. The principle of OAL is analyzed by this case.

Skywalk-04: Extending Metric monitoring information shows how to add metrics to an existing Source class.

This time, the Source class and OAL lexical syntax keywords are directly defined.

Source and Scope Extension for New Metrics

Identify targets for increase

Parsing this article through Java ManagementFactory, you can determine that the monitoring metrics are “number of classes currently loaded,” “Number of classes unloaded,” and “Total number of classes loaded.

ClassLoadingMXBean classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();
// The number of classes currently loaded
int loadedClassCount = classLoadingMXBean.getLoadedClassCount();
// The number of uninstalled classes
long unloadedClassCount = classLoadingMXBean.getUnloadedClassCount();
// The total number of classes loaded
long totalLoadedClassCount = classLoadingMXBean.getTotalLoadedClassCount();
Copy the code

define`agent` 与 `oap server`Communication class

In apm – protocol/apm – network/SRC/main/proto/language – agent/JVMMetric proto agreement file to increase the following definition.

Running MVN clean package -dskiptests =true in apm-protocol/apm-network will generate a new related Java class, Org.apache.skywalking.apm.net work. The language. The agent. The v3. Class is the Class in our code, the actual operation.

message Class {
  int64 loadedClassCount = 1;
  int64 unloadedClassCount = 3;
  int64 totalLoadedClassCount = 2;
}

message JVMMetric {
    int64 time = 1;
    CPU cpu = 2;
    repeated Memory memory = 3;
    repeated MemoryPool memoryPool = 4;
    repeated GC gc = 5;
    Thread thread = 6;
    // Add the definition of Class to the JVM metric
    Class clazz = 7;
}
Copy the code

collect`agent`After the message is sent to`oap server`

Collect indicators related to Class

package org.apache.skywalking.apm.agent.core.jvm.clazz;

import org.apache.skywalking.apm.network.language.agent.v3.Class;

import java.lang.management.ClassLoadingMXBean;
import java.lang.management.ManagementFactory;

public enum ClassProvider {
    /** * instance */
    INSTANCE;

    private final ClassLoadingMXBean classLoadingMXBean;

    ClassProvider() {
        this.classLoadingMXBean = ManagementFactory.getClassLoadingMXBean();
    }
	
    // Build the class indicator information
    public Class getClassMetrics(a) {
        int loadedClassCount = classLoadingMXBean.getLoadedClassCount();
        long unloadedClassCount = classLoadingMXBean.getUnloadedClassCount();
        long totalLoadedClassCount = classLoadingMXBean.getTotalLoadedClassCount();
        returnClass.newBuilder().setLoadedClassCount(loadedClassCount) .setUnloadedClassCount(unloadedClassCount) .setTotalLoadedClassCount(totalLoadedClassCount) .build(); }}Copy the code

In the org. Apache. Skywalking. Apm. Agent. The core. The JVM. JVMService# run method, set the related parameters of the class to the JVM index class

    @Override
    public void run(a) {
        long currentTimeMillis = System.currentTimeMillis();
        try {
            JVMMetric.Builder jvmBuilder = JVMMetric.newBuilder();
            jvmBuilder.setTime(currentTimeMillis);
            jvmBuilder.setCpu(CPUProvider.INSTANCE.getCpuMetric());
            jvmBuilder.addAllMemory(MemoryProvider.INSTANCE.getMemoryMetricList());
            jvmBuilder.addAllMemoryPool(MemoryPoolProvider.INSTANCE.getMemoryPoolMetricsList());
            jvmBuilder.addAllGc(GCProvider.INSTANCE.getGCList());
            jvmBuilder.setThread(ThreadProvider.INSTANCE.getThreadMetrics());
            // Set the class indicator
            jvmBuilder.setClazz(ClassProvider.INSTANCE.getClassMetrics());
			// Put the JVM's metrics in the blocking queue
            / / org. Apache. Skywalking. Apm. Agent. The core. The JVM. JVMMetricsSender# run method, relevant information will be sent to the oap server
            sender.offer(jvmBuilder.build());
        } catch (Exception e) {
            LOGGER.error(e, "Collect JVM info fail."); }}Copy the code

create`Source` 类

public class DefaultScopeDefine {
    public static final int SERVICE_INSTANCE_JVM_CLASS = 11000;

    /** Catalog of scope, the metrics processor could use this to group all generated metrics by oal rt. */
    public static final String SERVICE_INSTANCE_CATALOG_NAME = "SERVICE_INSTANCE";
}
Copy the code

package org.apache.skywalking.oap.server.core.source;

import lombok.Getter;
import lombok.Setter;

import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SERVICE_INSTANCE_CATALOG_NAME;
import static org.apache.skywalking.oap.server.core.source.DefaultScopeDefine.SERVICE_INSTANCE_JVM_CLASS;

@ScopeDeclaration(id = SERVICE_INSTANCE_JVM_CLASS, name = "ServiceInstanceJVMClass", catalog = SERVICE_INSTANCE_CATALOG_NAME)
@ScopeDefaultColumn.VirtualColumnDefinition(fieldName = "entityId", columnName = "entity_id", isID = true, type = String.class)
public class ServiceInstanceJVMClass extends Source {
    @Override
    public int scope(a) {
        return SERVICE_INSTANCE_JVM_CLASS;
    }

    @Override
    public String getEntityId(a) {
        return String.valueOf(id);
    }

    @Getter @Setter
    private String id;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "name", requireDynamicActive = true)
    private String name;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_name", requireDynamicActive = true)
    private String serviceName;
    @Getter @Setter @ScopeDefaultColumn.DefinedByField(columnName = "service_id")
    private String serviceId;
    @Getter @Setter
    private long loadedClassCount;
    @Getter @Setter
    private long unloadedClassCount;
    @Getter @Setter
    private long totalLoadedClassCount;
}
Copy the code

Will be taken from`agent`To obtain the information to`SourceReceive`

In the org. Apache. Skywalking. Oap. Server analyzer. The provider. The JVM. JVMSourceDispatcher to modify as follows

    public void sendMetric(String service, String serviceInstance, JVMMetric metrics) {
        long minuteTimeBucket = TimeBucket.getMinuteTimeBucket(metrics.getTime());

        final String serviceId = IDManager.ServiceID.buildId(service, NodeType.Normal);
        final String serviceInstanceId = IDManager.ServiceInstanceID.buildId(serviceId, serviceInstance);

        this.sendToCpuMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getCpu());
        this.sendToMemoryMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getMemoryList());
        this.sendToMemoryPoolMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getMemoryPoolList());
        this.sendToGCMetricProcess(
            service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getGcList());
        this.sendToThreadMetricProcess(
                service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getThread());
        // Class indicator processing
        this.sendToClassMetricProcess(
                service, serviceId, serviceInstance, serviceInstanceId, minuteTimeBucket, metrics.getClazz());
    }

    private void sendToClassMetricProcess(String service,
            String serviceId,
            String serviceInstance,
            String serviceInstanceId,
            long timeBucket,
            Class clazz) {
        // Assemble the Source object
        ServiceInstanceJVMClass serviceInstanceJVMClass = new ServiceInstanceJVMClass();
        serviceInstanceJVMClass.setId(serviceInstanceId);
        serviceInstanceJVMClass.setName(serviceInstance);
        serviceInstanceJVMClass.setServiceId(serviceId);
        serviceInstanceJVMClass.setServiceName(service);
        serviceInstanceJVMClass.setLoadedClassCount(clazz.getLoadedClassCount());
        serviceInstanceJVMClass.setUnloadedClassCount(clazz.getUnloadedClassCount());
        serviceInstanceJVMClass.setTotalLoadedClassCount(clazz.getTotalLoadedClassCount());
        serviceInstanceJVMClass.setTimeBucket(timeBucket);
        // Send the Source object to SourceReceive for processing
        sourceReceiver.receive(serviceInstanceJVMClass);
    }
Copy the code

in`OAL`Lexical and grammatical definitions`Source`The relevant information

The oap – server/oal – grammar/SRC/main/antlr4 / org/apache/skywalking/oal/rt/grammar/OALLexer g4 to define the Class keyword

// Keywords

FROM: 'from';
FILTER: 'filter';
DISABLE: 'disable';
SRC_ALL: 'All';
SRC_SERVICE: 'Service';
SRC_SERVICE_INSTANCE: 'ServiceInstance';
SRC_ENDPOINT: 'Endpoint';
SRC_SERVICE_RELATION: 'ServiceRelation';
SRC_SERVICE_INSTANCE_RELATION: 'ServiceInstanceRelation';
SRC_ENDPOINT_RELATION: 'EndpointRelation';
SRC_SERVICE_INSTANCE_JVM_CPU: 'ServiceInstanceJVMCPU';
SRC_SERVICE_INSTANCE_JVM_MEMORY: 'ServiceInstanceJVMMemory';
SRC_SERVICE_INSTANCE_JVM_MEMORY_POOL: 'ServiceInstanceJVMMemoryPool';
SRC_SERVICE_INSTANCE_JVM_GC: 'ServiceInstanceJVMGC';
SRC_SERVICE_INSTANCE_JVM_THREAD: 'ServiceInstanceJVMThread';
SRC_SERVICE_INSTANCE_JVM_CLASS:'ServiceInstanceJVMClass'; // Add the Class keyword to the OAL lexical definition
SRC_DATABASE_ACCESS: 'DatabaseAccess';
SRC_SERVICE_INSTANCE_CLR_CPU: 'ServiceInstanceCLRCPU';
SRC_SERVICE_INSTANCE_CLR_GC: 'ServiceInstanceCLRGC';
SRC_SERVICE_INSTANCE_CLR_THREAD: 'ServiceInstanceCLRThread';
SRC_ENVOY_INSTANCE_METRIC: 'EnvoyInstanceMetric';
Copy the code

The oap – server/oal – grammar/SRC/main/antlr4 / org/apache/skywalking/oal/rt/grammar/OALParser g4 to add the Class keyword

source : SRC_ALL | SRC_SERVICE | SRC_DATABASE_ACCESS | SRC_SERVICE_INSTANCE | SRC_ENDPOINT | SRC_SERVICE_RELATION | SRC_SERVICE_INSTANCE_RELATION | SRC_ENDPOINT_RELATION | SRC_SERVICE_INSTANCE_JVM_CPU | SRC_SERVICE_INSTANCE_JVM_MEMORY |  SRC_SERVICE_INSTANCE_JVM_MEMORY_POOL | SRC_SERVICE_INSTANCE_JVM_GC | SRC_SERVICE_INSTANCE_JVM_THREAD | SRC_SERVICE_INSTANCE_JVM_CLASS |// Add the keyword defined in the lexical definition to the OAL syntax definitionSRC_SERVICE_INSTANCE_CLR_CPU | SRC_SERVICE_INSTANCE_CLR_GC | SRC_SERVICE_INSTANCE_CLR_THREAD | SRC_ENVOY_INSTANCE_METRIC  | SRC_BROWSER_APP_PERF | SRC_BROWSER_APP_PAGE_PERF | SRC_BROWSER_APP_SINGLE_VERSION_PERF | SRC_BROWSER_APP_TRAFFIC | SRC_BROWSER_APP_PAGE_TRAFFIC | SRC_BROWSER_APP_SINGLE_VERSION_TRAFFIC ;Copy the code

Running MVN clean package -dskiptests =true in the oap-server/oal-grammar directory will generate a new related Java class

define`OAL`indicators

The oap – server/server – the bootstrap/SRC/main/resources/oal/Java – agent. Add based on oal oal related parameters of the Class definition of grammar

// The number of classes currently loaded
instance_jvm_class_loaded_class_count = from(ServiceInstanceJVMClass.loadedClassCount).longAvg();
// The number of uninstalled classes
instance_jvm_class_unloaded_class_count = from(ServiceInstanceJVMClass.unloadedClassCount).longAvg();
// The total number of classes loaded
instance_jvm_class_total_loaded_class_count = from(ServiceInstanceJVMClass.totalLoadedClassCount).longAvg();
Copy the code

configuration`UI`panel

Import the following configuration to the APM panel

{
  "name": "Instance"."children": [{
      "width": "3"."title": "Service Instance Load"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "service_instance_cpm"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "CPM - calls per minute"
    },
    {
      "width": 3."title": "Service Instance Throughput"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "service_instance_throughput_received,service_instance_throughput_sent"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "Bytes"
    },
    {
      "width": "3"."title": "Service Instance Successful Rate"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "service_instance_sla"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "%"."aggregation": "/"."aggregationNum": "100"
    },
    {
      "width": "3"."title": "Service Instance Latency"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "service_instance_resp_time"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "ms"
    },
    {
      "width": 3."title": "JVM CPU (Java Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_jvm_cpu"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "%"."aggregation": "+"."aggregationNum": ""
    },
    {
      "width": 3."title": "JVM Memory (Java Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_jvm_memory_heap, instance_jvm_memory_heap_max,instance_jvm_memory_noheap, instance_jvm_memory_noheap_max"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "MB"."aggregation": "/"."aggregationNum": "1048576"
    },
    {
      "width": 3."title": "JVM GC Time"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_jvm_young_gc_time, instance_jvm_old_gc_time"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "ms"
    },
    {
      "width": 3."title": "JVM GC Count"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."queryMetricType": "readMetricsValues"."chartType": "ChartBar"."metricName": "instance_jvm_young_gc_count, instance_jvm_old_gc_count"
    },
    {
      "width": 3."title": "JVM Thread Count (Java Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."metricName": "instance_jvm_thread_live_count, instance_jvm_thread_daemon_count, instance_jvm_thread_peak_count,instance_jvm_thread_deadlocked,instance_jvm_thread_monitor_deadlocked"
    },
    {
      "width": 3."title": "JVM Thread State Count (Java Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_jvm_thread_new_thread_count,instance_jvm_thread_runnable_thread_count,instance_jvm_thread_blocked_thread_count ,instance_jvm_thread_wait_thread_count,instance_jvm_thread_time_wait_thread_count,instance_jvm_thread_terminated_thread_ count"."queryMetricType": "readMetricsValues"."chartType": "ChartBar"
    },
    {
      "width": 3."title": "JVM Class Count (Java Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_jvm_class_loaded_class_count,instance_jvm_class_unloaded_class_count,instance_jvm_class_total_loaded_class_cou nt"."queryMetricType": "readMetricsValues"."chartType": "ChartArea"
    },
    {
      "width": 3."title": "CLR CPU (.NET Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_clr_cpu"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "%"
    },
    {
      "width": 3."title": "CLR GC (.NET Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_clr_gen0_collect_count, instance_clr_gen1_collect_count, instance_clr_gen2_collect_count"."queryMetricType": "readMetricsValues"."chartType": "ChartBar"
    },
    {
      "width": 3."title": "CLR Heap Memory (.NET Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."metricName": "instance_clr_heap_memory"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."unit": "MB"."aggregation": "/"."aggregationNum": "1048576"
    },
    {
      "width": 3."title": "CLR Thread (.NET Service)"."height": "250"."entityType": "ServiceInstance"."independentSelector": false."metricType": "REGULAR_VALUE"."queryMetricType": "readMetricsValues"."chartType": "ChartLine"."metricName": "instance_clr_available_completion_port_threads,instance_clr_available_worker_threads,instance_clr_max_completion_port_t hreads,instance_clr_max_worker_threads"}}]Copy the code

Results the check

You can see that in the import interface, there are already indicators related to Class

Code contributions

Add some new thread metric and class metric to JVMMetric #7230
add some new thread metric and class metric to JVMMetric #52
Remove Terminated State and New State in JVMMetric (#7230) #53
Add some new thread metric and class metric to JVMMetric (#7230) #7243

Reference documentation

Observability Analysis Language
Source and Scope extension for new metrics
Java ManagementFactory parsing

Share and record what you learn and see

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Skywalk-06: OAL Basics

`OAL`Basic knowledge of

Basic introduction

grammar

The grammar of case

Start with a case study`OAL`The principle of

Missing monitoring of class loading information

Identify targets for increase

define`agent` 与 `oap server`Communication class

collect`agent`After the message is sent to`oap server`

create`Source` 类

Will be taken from`agent`To obtain the information to`SourceReceive`

in`OAL`Lexical and grammatical definitions`Source`The relevant information

define`OAL`indicators

configuration`UI`panel

Results the check

Code contributions

Reference documentation

Skywalk-06: OAL Basics

OALBasic knowledge of

Basic introduction

grammar

The grammar of case

Start with a case studyOALThe principle of

Missing monitoring of class loading information

Identify targets for increase

defineagent 与 oap serverCommunication class

collectagentAfter the message is sent tooap server

createSource 类

Will be taken fromagentTo obtain the information toSourceReceive

inOALLexical and grammatical definitionsSourceThe relevant information

defineOALindicators

configurationUIpanel

Results the check

Code contributions

Reference documentation

Related Posts

Dubbo (4) : Core source code – Cut to Spring

Interviewer: Tell me about the design idea of seckill system in high concurrency scenario

Spring Transaction Management

`OAL`Basic knowledge of

Start with a case study`OAL`The principle of

define`agent` 与 `oap server`Communication class

collect`agent`After the message is sent to`oap server`

create`Source` 类

Will be taken from`agent`To obtain the information to`SourceReceive`

in`OAL`Lexical and grammatical definitions`Source`The relevant information

define`OAL`indicators

configuration`UI`panel