background

In order to better actively perceive the PK indicators of user-level rival products, for example, is Huobi opened faster or Binan opened faster? Is the Huobi WebSocket connection faster or Binan connection faster?

Now these are the questions, it is very important to a woodpecker perception map is competing goods data contrast, now we rely on third-party testing platform (such as: Boce, listen to the cloud), but the data is not good to our needs, and the data must be the most safety directly in our own hands, is the so-called “the data in hand, all I have.”

The target

  • Docker the deployment
  • Plug-in management
  • Scheduled collection of tasks
  • Custom metrics

Architecture diagram

Runtime environment

Run based on Docker and run as Node.js, you can do:

  1. Collect data based on Puppeteer headless browsers such as performance metrics, WebSocket metrics, interface metrics, etc
  2. Completed based on Ping: domain availability, TTL, packet loss rate detection

Purchase multiple node servers such as: Woodpecker Docker Agent is installed on AliCloud ECS 1U1G nodes in Beijing, Hangzhou, Shanghai, etc., which can actually be understood as the concept of “chicken”. We manage the plug-ins and state installed by these agents in the remote Master, and send them to the corresponding node when the task is triggered. Data collection and reporting are completed by nodes.

It should be noted that the Agent only runs a Docker container in the ECS, and the Master manages/operates the same container. It does not operate on the host and does not overstep its authority (e.g., SSH ECS, which we consider to be the scope of OPS processing).

Master

1. Plug-in management

In the form of code (Node.js), it is managed by modules. Select plug-ins in the task and configure the required parameters of plug-ins. The list of plug-ins is as follows:

  1. Ping plug-in

    • Configuration: domain name /IP, timeout, times

  2. Web page performance metrics plug-in

    • Configuration: link, view width and height, UserAgent
  3. Huobi Global WebSocket test
  4. Binance WebSocket test
  5. There’s a lot more you can do…
2. The Agent management

Check the status and distribution of all agents, install/uninstall plug-ins, and manage regional distribution, label and other information.

3. Task management

Send data to the corresponding Agent in a task queue. The task configuration is roughly as follows:

  • The name of the task
  • Agent, such as BJ01, SG03, HZ03
  • Interval of tasks: single, 1s, 1m, 1h, 1h

  • Execution plug-in:

    1. Ping – { url: ‘www.baidu.com’, timeout: 1000, count: 5 }
    2. Web Performance – { url: ‘https://www.huobi.me/zh-cn/ex…’ }
    3. Web Performance – { url: ‘https://www.baidu.com’ }
    4. Global WebSocket
    5. Binance WebSocket

Agent

Run a container in the background using Docker, such as:

Docker run \\ -d \\ --name woodpecker-agent \\ -p 8080:8080 \\ -e ACCESS\_KEY\= Generated token \\ Hub. The Woodpecker. The VIP/pk/agent: v1.0.0

OpenAPI provided by this Agent:

  • IP:8080/open/v1/status – current Agent status
  • IP: 8080 / open/v1 / plugins/install – installing a plug-in, plug-in data uploading to the Agent
  • IP: 8080 / open/v1 / task/exec – use the data to perform a task

Woodpecker

Log collection service is pushed to the service by Agent after collection, and then the service enters the database.

Pay attention to

  1. All above use internal network transmission, such as: master-> agent-> woodpecker-> DB
  2. Frequent collection of third-party websites may face the risk of IP blocking