This is the second day of my participation in the August More Text Challenge

【 Basic Introduction 】

  • Esper is an event flow processing (ESP) and event correlation engine (CEP, complex event processing).
  • Esper is targeted at real-time Event-driven architecture (EDA).
  • Custom actions written by PlainOldJavaObjects (POJO) are triggered when Esper detects the occurrence of a qualified time in the event flow.

When millions of events occur simultaneously, it is impossible to use a normal relational database to store and query them, and Esper is designed for such large numbers of correlated events.

【Esper 】

  • Esper provides a custom event processing language (EPL) that allows the conditional expression of rich events, correlations, and possibilities across time Windows, thus reducing the development effort required to set up a system that can respond to complex situations.
  • Esper is a lightweight Java-written kernel that is fully embedded into any Java process, JEE application server or Java-based Enterprise Service Bus. It enables the application to receive a large number of messages or events during the process of rapid development.

Introducing the flow of events and complex event information is critical to making informed decisions.

Esper, I have to say CEP. CEP stands for Complex Event Process.

It sounds complicated, but in fact it is based on the event stream, abstracting the data to be analyzed into events, and then sending the data to the CEP engine, which will get the event processing results based on the input of the event and the originally registered processing model.

【 INTRODUCTION to CEP 】

Complex Event Processing

Generally speaking, events refer to what is happening in a system, which may occur at all levels of the system. It can be an action, such as customer placing an order, sending a message, submitting a report, etc., or a change in a certain state, such as temperature change, timeout, etc.

Through the analysis of these events, the effective information can be extracted. According to Wikipedia, Event processing refers to tracking events that occur in a system, analyzing the information in the Event and drawing conclusions from it. Complex event processing, on the other hand, combines events from multiple event sources to infer events in more complex situations.

Thus, the purposes of CEP include:

  • (1) Identify the events needed;
  • (2) Fast processing of these events.

Usually, what we want to achieve with CEP is to grasp the current situation or state of affairs. Therefore, CEP is not interested in the information given by the event itself, but in the conclusion that can be deduced from the information. Through CEP, we can make sense of the event.

To implement a CEP engine, some things to consider include:

  • (1) Throughput;
  • (2) Low latency, from the event to the event is processed, there cannot be too much delay;
  • (3) Complex logic processing, CEP needs to be able to carry out relatively complex operations on events, such as detecting the correlation between events, filtering, adding Windows, connecting, etc.

CEP is a standard, and Esper is just an open source implementation of it. In addition to Esper, many large companies have similar business software, such as IBM, Sybase, etc., which is said to be extremely expensive.

An important feature of CEP is that it is an in-memory calculation tool and an SQL-like statement. In-memory computing can be a double-edged sword.

  • Benefits from needless to say, in a word: quick! The downside is obvious: there is a risk of data loss, and there are capacity limitations (real-time computing is not really limited by the size of memory, but by how real-time is defined, i.e., the specific business).
  • So if the business can’t tolerate data loss, then the high availability solution has to be good, and Esper’s high availability is tough.

CEP’s SQL-like statements can be understood as definitions and descriptions of processing models. This is a special statement that runs in the CEP engine. It is called SQL-like because it is exactly like SQL, except select, INSERT, delete, update, and avG, count, etc.

So for those of you who know SQL, the syntax structure is pretty predictable. In Esper, this sentence is called EPL, or Event Process Language. As the core of Esper.

The custom event processing language EPL allows the expression of rich event conditions, associations, and across event Windows, minimizing the development difficulty of having to build systems to deal with complex situations.

【 summary 】

Esper is a lightweight engine for the Java kernel for event handling and complex event correlation handling. For the processing of a large number of events, can use the shortest time to respond to trigger the corresponding operation. In addition, an event processing language tailored for Esper, the SQL-like language, is very convenient for the expression of event conditions and relational processing.

Flow of events and complex events

The stream of events — an infinite collection of events — can be thought of as a sliding window, where further correlation makes a lot of sense, and responding to events with minimal delay is essential for effective action and competitive advantage.

The stream of events is actually a series of discrete events. In reality, the monitoring or processing of event flow, such as stock information, needs to calculate its trend in real time so as to obtain valuable data in time. Event flow is a sliding window in Esper that is critical for event correlation processing and so on.

Esper

Relational databases or message-based systems such as JMS can be really difficult to handle temporal data and real-time queries. In fact, the database requires an explicit query to return meaningful data, and it is not appropriate to push data when it is released to change. JMS systems are stateless and require developers to implement the timing and aggregation logic themselves. Esper, by contrast, offers a higher level of abstraction and intelligence that can be thought of as an inversion of the database: Instead of storing data, Esper allows applications to store queries and let the data run through.

Esper responds in real time when conditions that meet the query occur. The execution module is continuous, not only when the query is submitted. This concept is a key foundation of EDA (Event-driven Architecture) and has been under active research for the past decade or more. However, real recognition of the importance of the system has only recently begun to emerge.

In Esper, a custom EPL allows queries to be registered in the engine. Listener classes (POJO classes) are called by the engine when events meet EPL criteria. EPL can express complex matching conditions, including event Windows, event flow association, filtering, aggregation and sorting, etc. The Esper statement can also be followed by a “->” condition to retrieve complex events from multiple simple events. Events can be described as

JavaBean classes, traditional Java classes, XML, or Maps, as message publishers, promote reuse of existing systems.

【 summary 】 in dealing with real-time data, such as, the stock price movements, etc., with a heart rate monitor Esper relative to the relational database has natural advantages, do not need to complete data storage Esper, can complete the data real-time query processing, seen in this way, more like a database of inverted – imagine a database is to store data first, SQL is compiled and parsed to complete the query of stored data. Esper is compiled first

EPL statement, the formation of a filter (or processing) layer (or network), real-time data, through the filter layer to complete the effective event screening or formation of effective events. Esper’s event processing language, EPL, is a SQL-like language, and its SELECT statement, FROM statement, WHERE statement, group by, HAVING, and even most of the aggregate functions are consistent with the standard SQL language. In EPL rule writing

For those who have database development experience, it is very easy to get started.

4. Develop event-driven applications

Developing event-driven applications with Esper is not difficult. There are probably the following steps:

  • 1> Define the functions or tasks that the application should perform by analyzing the business domain and defining the scenarios to be explored or the information to be reported.
  • 2> Define performance requirements, especially throughput and latency.
  • 3> Determine where to get events from.
  • 4> Determine event format and event content based on business.
  • 5> Design event relationships that lead to complex events.
  • 6> Event source.
  • 7> Design event presentation types: Java class, Map, or XML, etc
  • 8> Define EPL statements for matching pattern and stream processing.
  • 9> For example, use the CSV adapter as an event simulation tool to test test scenarios, or generate load load events.
  • 10> On the final environment, test throughput and latency.

Design the event representation

In Esper, Java classes are a simple, rich, and versatile way to represent events. Java classes provide inheritance and polymorphism through interfaces and superclasses, and a complex business domain can be represented by a single object, Graph. Maps or XML are also a way to represent events.

Event flow analysis

EPL statements derive and aggregate information from one or more streams of events, join or merge streams of events, and feed the results of a time stream to subsequent statements.

EPL is similar to SQL in its use of SELECT and WHERE statements.

But the EPL statement replaces the database tables with event streams and views. Similar to tables in SQL, views define data that can be queried and filtered. Views can be represented as a window on an event stream.

Views can also sort events, get statistics from event attributes, group events, or handle unique event attribute values.

The following EPL statement is used to calculate the average price of a stock event over the last 30 seconds:

Select avg(price) from stocktickevent. win:time(30 SECCopy the code

Here EPL returns the average of the top 100 stock events:

The select symbol, avg (price) as averagePrice from StockTickEvent. Win: length (100) group by symbol duplicate codeCopy the code

The following example relates two event flows.

The first event stream consists of fraud alarm events and stores the event information of the last 30 seconds.

The second stream is an WITHDRAWAL event that holds events from the last 30 seconds. These two flows are related by account number:

select fraud.accountNumber as accntNum, fraud.warning as warn, withdraw.amount as amount,MAX(fraud.timestamp, withdraw.timestamp) as timestamp, 'withdrawlFraud' as desc from FraudWarningEvent.win:time(30 min) as fraud, Withdrawalevent.win :time(30 SEC) as withdraw where fraud.accountNumber = withdraw. AccountNumberCopy the code

Event pattern matching

Event patterns match when an event or multiple events occur that match the pattern’s definition.

When one or more events occur, the event pattern will match the defined pattern. Patterns can also be time-based. Pattern matching is achieved through state machines.

Pattern expressions can consist of filter expressions that are concatenated with pattern operators. Deeper nesting of pattern expressions can be achieved by embedding expressions in parentheses.

Class 5 operators:

Β· Control the creation and termination of mode detectors: every

Β· Logical operators: AND, OR, not

Β· Time operator to operate on the sequence of events: -> (followed by, followed by)

Β· Filter the where condition of the output event, causing the termination of the mode detector: e.g., Timer :within

Β· Observers observe time events like other events, such as timer:interval, timer:at

Here are some examples of EPL:

Every StockTickEvent(symbol="IBM", price>80) where timer:within(60 seconds) copies the codeCopy the code
// Every timer:at(5, *, *, *, *) copy code at the 5th minute of every hourCopy the code
A -> (B or C) copy code if A occurs and B or C followsCopy the code
If the EventY event is followed by an EventY event and its objectID is the same as EventX's objectID, a reminder will be given (output an event) : Every A =EventX -> every B =EventY(objectID= a.objectid) Copy the codeCopy the code

Used in combination with pattern matching and event flow analysis

When a sequence of events is detected (or no event occurs), the pattern matches. The results of pattern matching can be used for further analysis and processing.

The following mode monitors the scenario where no Status event with the same ID occurs 10 seconds after the Status event occurs. The entire EPL statement further counts the total number of events per ID that all occur.

Select a.id, count(*) from pattern [every a=Status -> (timer:interval(10 SEC) and not Status(id=a.id)] group by idCopy the code

Named window

Named Windows are global data Windows in the engine that can participate in many statement queries and can be used by multiple statements to perform SELECT, INSERT, and DELETE operations. Named Windows are similar to tables in a relational database system.

To create a named window, do the following:

Create Window AlertNamedWindow AS (Origin String, Priority String, alarmNumber Long) Copy codeCopy the code

When the event arrives, a SELECT, UPDATE, or DELETE operation can be triggered. Here is a select application, a simple count of the total number of rows in the data window:

On TriggerEvent select count(*) from AlertNamedWindow Copies codeCopy the code

Matching Pattern Matching (Matching Pattern Matching)

Matching – recognition pattern is a pattern matching syntax based on regular expressions and is recommended as a standard syntax for SQL.

The matching and recognition patterns below detect patterns that are likely to occur in events that are saved through named Windows (as declared above). This pattern looks for two events that follow closely, that is, there is no same origin between the events. The first event must have a “high” priority, and the second event must have a “medium” priority.

select * from AlertNamedWindow match_recognize ( partition by origin measures a1.origin as origin, a1.alarmNumber as alarmNumber1, a2.alarmNumber as alarmNumber2 pattern (a1 a2) define a1 as a1.priority = 'high', A2 as A2. Priority = 'medium') copies the codeCopy the code

Variables

A variable is a scalar, object, or event value that can be used in all statements, including schemas. Variables can be used in expressions anywhere in the EPL.

【 Summary 】 When developing Esper applications, Java classes are recommended for event types, which are more intuitive for the description of event information (see Quick Start). The Map approach is not recommended for event design with complex relationships (see the performance section of the Esper Reference documentation). In the design of EPL, according to business requirements, matching Pattern should not be used if it can be completed by standard SQL syntax, because extra parsing of Pattern is required at runtime, and its rules are more complex than SQL, resulting in a little performance loss.

The use of data Windows allows Esper to handle more complex application scenarios, such as working with distributed caches and static data. Variables are not hard to understand. They are used in advanced development languages such as Java, C/C++, scripting languages such as Ruby, JS, etc., and their use is limited to the current Esper engine instance.