preface

This example describes the application of YoMo in industrial Internet data collection. Taking the collection of noise sensor data as an example, it involves the whole process of data collection/processing/workflow/data display. In order to experience the operation effect, it will be containerized and the experience version will be quickly deployed through Docker.

predicate

  • Xxx-source: represents a data source collector
  • Xxx-zipper: indicates a workflow and control plane
  • Xxx-flow: indicates a workflow unit that is used for actual business logic processing and is scheduled by Zipper.
  • Xxx-sink: indicates the transmission destination of a data. Generally, the data is sent to the database or to the next agent and is scheduled by zipper.

architecture

As can be seen from the figure, two independent regions, the edge end and the cloud, are separated through weak network or Internet connection. Here is a brief introduction of each service:

  • Sensor device (Noise) and data collection gateway (ZLAN) are deployed at the edge. The gateway will periodically request the status data to the device and convert it into MQTT protocol data and send it to the noise-Source collector. The source plays the role of converting the code and establishing the connection with the YoMo workflow engine Zipper. Hardware selection and configuration for sensor devices and data collection gateways can be included in this article: yomo.run/zh/ AIot.
  • For developers who don’t want to buy hardware, a Noise-Emitter simulator is available to generate noisy data.
  • Zipper is a powerful workflow engine that schedules multiple flows and sinks through orchestration (workflow.yaml), allowing them to stream business logic together to meet complex requirements. QUIC+Y3 is used for all communications, codecs and codecs connected with it, providing reliable real-time streaming processing and experiencing the fun of streaming programming throughout the whole process.
  • Noise-flow implements a simple process of dividing the noise value by 10, and monitors output logs to alert if a certain threshold is exceeded.
  • Noise-sink does not actually output to the database, but builds a WebSocket server to output real-time noise status to any web page for display and consumption.
  • Noise-web is a web service that consumes WebSocket. It can be deployed anywhere, as long as it can access the WebSocket service address provided by noise-sink. Here we assume that deployment back to the edge is also ok.

code

The following table provides all the code of the case for interested friends to view, referring to the code of this case body, you can easily develop the case of similar scenarios.

project address instructions
noise-source yomo-source-noise-example Collect noise data in MQTT message format
noise-zipper yomo-zipper-noise-example Orchestrate the workflow and data flow of the case body
noise-flow yomo-flow-noise-example Noise data is pre-processed and alerted
noise-sink yomo-sink-socketio-server-example Provide WebSocket service for data presentation
noise-web yomo-sink-socket-io-example Consuming the WebSocket service displays the noise state
noise-emitter yomo-source-noise-emitter-example Simulation produces noisy data
quic-mqtt yomo-source-mqtt-starter Develop common components for XXX-source

Containerized deployment

On the download section of the project code can undertake local native deployment, experience the delight of YoMo development, but for the friend that want to eager to see the effect immediately, in a better way, of course, is the first to look at the effect of the fast running, so the last project has also done a container, the root directory of every project provides Dockerfile file, And the official image is available for download at hub.docker.com:

project Mirror address The latest version
noise-source yomorun/noise-source yomorun/noise-source:latest
noise-zipper yomorun/noise-zipper yomorun/noise-zipper:latest
noise-flow yomorun/noise-flow yomorun/noise-flow:latest
noise-sink yomorun/noise-sink yomorun/noise-sink:latest
noise-web yomorun/noise-web yomorun/noise-web:latest
noise-emitter yomorun/noise-emitter yomorun/noise-emitter:latest
quic-mqtt yomorun/quic-mqtt yomorun/quic-mqtt:latest

Yomorun/QUIC-MQTT: Latest is the base image for developing XXX-Source and can be quickly packaged with custom code, but can be ignored in this case.

Rapid deployment

To get a quick run-time experience, this section describes how the entire container runs on the same host.

fast

With the official image above, it’s much easier to experience the effect in just a few simple steps:

  • Download the docker-comemage. yml file.
  • rundocker-compose up -d
  • Have a cup of tea and wait. Visit http://localhost:3000/ to see the following image:

Matters needing attention:

  • The Delay value here may not be very accurate, because it is deployed through the Docker container, and the clocks of each container are not perfectly aligned. To view the most accurate Delay value, noise-Source and noise-Web should be deployed natively on the same host.
  • If the deployment is not local, change the environment variable SOCKET_SERVER_ADDR in the docker-comemess. yml file to the host address of the service you are deploying.

Check the status

Docker-compose PS to check service status

Name Command State Ports ----------------------------------------------------------------------------------------- noise-emitter sh -c go run main.go Up noise-flow sh -c yomo run app.go -p 4242 Up 4242/udp noise-sink sh -c go run main.go Up 4141/udp, 0.0.0.0:8000->8000/ TCP noise-source sh -c go run main.go Up 1883/ TCP noise-web./docker-entrypoint. Up 0.0.0.0:3000->3000/ TCP noise-zipper sh -c yomo wf run workflow... Up 9999/udpCopy the code
  • Noise-sink exposes 8000 WebSocket ports for noise-Web display consumption.
  • Noise-web exposes 3000 HTTP ports for displaying real-time noise values and latency.
  • Noise-zipper /noise-flow/noise-sink all provide UDP port QUIC service, the whole process of QUIC communication.
  • Noise-source is our key for docking different devices and provides an MQTT port of 1883, which can also be modified.

See the log

  • View noise-emitter docker-compose logs -f noise-emitter

    noise-emitter    | 2021-04-26 10:11:03: Publish counter=12438, topic=NOISE, payload={"noise":12438}
noise-emitter    | 2021-04-26 10:11:04: Publish counter=12439, topic=NOISE, payload={"noise":12439}
noise-emitter    | 2021-04-26 10:11:05: Publish counter=12440, topic=NOISE, payload={"noise":12440}
noise-emitter    | 2021-04-26 10:11:06: Publish counter=12441, topic=NOISE, payload={"noise":12441}

Copy the code

    This simulation generator generates MQTT data: the subject is NOISE and the value is an increasing serial number (JSON format).

  • Check noise-source docker-compose logs -f noise-source

    noise-source     | 2021/04/26 15:27:32 receive: topic=NOISE, payload={"noise":2638}
noise-source     | 2021/04/26 15:27:32 write: sendingBuf=[]byte{0x81, 0x1b, 0x90, 0x19, 0x11, 0x3, 0x45, 0x24, 0xe0, 0x12, 0x6, 0xaf, 0x90, 0xe8, 0xce, 0x83, 0x1c, 0x13, 0xa, 0x31, 0x37, 0x32, 0x2e, 0x31, 0x39, 0x2e, 0x30, 0x2e, 0x36}
noise-source     | 2021/04/26 15:27:33 receive: topic=NOISE, payload={"noise":2639}
noise-source     | 2021/04/26 15:27:33 write: sendingBuf=[]byte{0x81, 0x1b, 0x90, 0x19, 0x11, 0x3, 0x45, 0x24, 0xf0, 0x12, 0x6, 0xaf, 0x90, 0xe8, 0xce, 0x8b, 0x5, 0x13, 0xa, 0x31, 0x37, 0x32, 0x2e, 0x31, 0x39, 0x2e, 0x30, 0x2e, 0x36}

Copy the code
    • Receive: indicates the MQTT data received by source.
    • Write: Y3 transcoded bytecode is sent to noise-Zipper workflow engine.

  • Check noise-zipper docker-compose logs -f noise-zipper

    noise-zipper | 2021/04/26 14:39:28 Found 1 flows in zipper config noise-zipper | 2021/04/26 14:39:28 Flow 1: Noise Serverless on noise-flow:4242 noise-zipper | 2021/04/26 14:39:28 Found 1 sinks in zipper config noise-zipper | 2021/04/26 14:39:28 Sink 1: Socket.io Server on noise-sink:4141 noise-zipper | 2021/04/26 14:39:28 Running YoMo workflow... Noise - zipper | 2021/04/26 14:39:28 ✅ Listening on 0.0.0.0:9999 noise - zipper | 2021/04/26 14:43:32 ✅ Connect to noise Serverless (4242), whose - flow: successfully. The noise - zipper | 2021/04/26 14:44:33 ✅ Connect to Socket. IO Server (noise-sink:4141) successfully.Copy the code

    The workflow engine connects the noise-flow and noise-sink workflow processing units.

  • View noise-flow docker-compose logs -f noise-flow

    Noise - flow | ❗ value: 561.700012 reaches the threshold 16! 45.700012 noise - flow | [172.19.0.6] 1619425035923 > value: 561.799988 ⚡ ️ = 0 ms noise - flow | ❗ value: 561.799988 REACHES the Threshold 16! 45.799988 noise - flow | [172.19.0.6] 1619425036923 > value: 561.900024 ⚡ ️ = 1 msCopy the code

    The noise data is generated by the simulator and far exceeds the preset threshold, printing a warning message.

Partition deployment

After the rapid deployment in the previous section, we should know how to conduct containerized deployment on the same host and check the status of each service. However, this is not consistent with the actual architecture diagram of separation of cloud and edge proposed at the beginning. Serverless services such as Flow /sink will be deployed in the cloud in realistic scenarios. The data sink source will be deployed on the edge, so this section will isolate them and see how to orchestrate the deployment.

The cloud deployment

Deploy the service

The goal is to deploy noise-zipper/noise-flow/noise-sink in the cloud. You can view the following configuration files and running steps:

  • Download the docker-comement-cloud. yml file.
  • rundocker-compose -f docker-compose-cloud.yml up -d

Check the status

Run docker-comement-f docker-comement-cloud. yml ps to check the service status

Name Command State Ports ---------------------------------------------------------------------------------------- noise-flow sh -c yomo run app.go -p 4242 Up 4242/udp noise-sink sh -c go run main.go Up 4141/udp, 0.0.0.0:8000->8000/ TCP noise-zipper sh -c yomo wf run Workflow... The Up 0.0.0.0:9999 - > 9999 / udpCopy the code
  • Noise-zipper exposes the service port of the cloud workflow engine, which is a UDP port since the communication is QUIC protocol.
  • Noise-sink exposes ports for data consumption through WebSocket, allowing other Web services to display consumption.

Edge side deployment

Deploy the service

The goal is to deploy noise-source/noise-web/noise-Emitter on the edge. You can view the following configuration files and running steps:

  • Download the docker-comement-edge. yml file.
  • rundocker-compose -f docker-compose-edge.yml up -d

Matters needing attention:

  • The SOCKET_SERVER_ADDR variable in docker-comement-edge. yml needs to be set to the address and port exposed by source-sink on the cloud side (default 8000).

Check the status

Run docker-comement-f docker-comement-edge. yml ps to check the service status

Name Command State Ports ------------------------------------------------------------------------------- noise-emitter sh -c go run main.go Up noise-source sh -c go run main.go Up 1883/tcp noise-web docker-entrypoint.sh sh -c ... The Up 0.0.0.0:3000 - > 3000 / TCPCopy the code
  • Noise-web exposes the port of the effect presentation site. Visit http://localhost:3000/ to see the same screen as before.

Router Deployment

And that’s where it ends? Also have no! In fact, edge deployment is more complicated than cloud deployment because many edge devices are older and do not necessarily support Docker containers, so you can choose to compile the source code into binaries for different platforms and run noise-Source directly on the corresponding platform. Of course, a more convenient choice is to buy a router supporting Docker container, which can be directly deployed in the edge end through the container. At present, YoMo has reached in-depth cooperation with iKuai to jointly promote the application and development of industrial Internet in various fields. You can view the detailed information through the following link:

  • IKuai +YoMo accelerates the cloud native era of industrial Internet
  • IKuai +YoMo edge computing empowers industrial Internet

Refer to the reference

This is all the code and deployment of the noise sensor case from data collection and processing to presentation. For developers who need to expand or extend to other application scenarios, you can click on the link of each project for detailed reading. Each project is microservialized construction, service roles are clear, and the code is clear and easy to understand. If you have any questions, please raise Issues or discuss. Reference links:

  • yomo.run/
  • Github.com/yomorun/yom…
  • Github.com/yomorun/exa…

Finally, add an architecture diagram with exposed port numbers for each service for those who are in trouble.