My latest and most complete articles are in the pumpkin slow say www.pkslow.com, welcome to tea!

1 introduction

GCP’s Pubsub is an asynchronous messaging service that isolates services that produce events from services that process them. Message queues are of little use, similar to Kafka, RabbitMQ, etc. An important reason for using Pubsub is that it is not necessary to manage the operation and maintenance of the entire middleware itself, but to leave the professional work to the professional team. In this way, it is also a way to save costs.

GCP also offers Pubsub Lite at a lower cost, which I won’t cover here.

2 the concept

2.1 Basic Concepts

Some important core concepts:

  • Topic: a resource to which a producer sends a message;
  • Subscription: message flow resources for a single specific topic. Each subscription belongs to a topic and is subscribed to any topic of interest.
  • 1. Transmitted data and characteristics;
  • Publisher: also called producer, responsible for sending messages to topics;
  • Subscriber: Also called consumer, responsible for reading messages from the subscription.

Their relationship: A publisher sends a message to a particular topic, and there are one or more subscriptions under the topic, from which the subscriber reads the message. So the relationship between publisher and subscriber is as follows:

  • One to one.
  • A couple more;
  • For a more;
  • Many to many.

As shown below:

2.2 Message transfer process

The entire message delivery process is roughly as follows:

(1) The publisher sends a message to the topic containing the data to be sent and the characteristics of the message;

(2) After the system receives the message, it will be saved in the GCP system;

(3) The system forwards the message of the topic to the subscription;

(4) Pubsub pushes messages to subscribers, or subscribers pull messages;

(5) After receiving the information, subscribers will return Ack confirmation information;

(6) Pubsub removes confirmed messages.

2.3 Integrating Other Components

The entire Stack of GCP can be integrated with each other, and other components such as Pubsub can be integrated as follows:

3 Pubsub Quick start

3.1 Using the Gcloud Command Line

The following is how to get started using the SDK command line tool Gcloud:

# create theme
$ gcloud pubsub topics create pkslow-topic

Create a subscription
$ gcloud pubsub subscriptions create pkslow-sub --topic=pkslow-topic

# Post a message
$ gcloud pubsub topics publish pkslow-topic --message="www.pkslow.com"

# Receive message
$ gcloud pubsub subscriptions pull pkslow-sub --auto-ack
Copy the code

3.2 Using a Client library

Pubsub supports a wide range of languages, including Python, Java, C++, Go, node.js, PHP, etc., which can be used by general projects. The previous article, “Integrating Spring Cloud Stream Binder with GCP Pubsub for message sending and receiving,” covered Java integration, which I will not cover here.

4 Sorting Messages

Message sorting is A useful feature that ensures the order of messages, that is, if the publisher sends the message A-B-C-D, it should receive it A-B-C-D, not A-B-D-C or something else. The ordering of Pubsub messages requires the cooperation of both publishers and subscribers:

(1) The publisher must specify the Ordering Key when sending messages. This Key does not tell Pubsub which messages should be ordered, but which messages should be ordered. All messages with the same Key are sorted by time. Messages with different keys are not ordered and do not need to be sorted. So, specify the same Key for messages that need to be chronological.

(2) The Ordering feature must be enabled for subscribing, otherwise the messages will not be ordered even if the Ordering Key is configured.

The sorting feature is useful, but it hurts performance.

Some difficulties encountered:

(1) For Java development, Spring Cloud Stream does not support Pubsub sorting, so it needs to use Google’S SDK to develop or modify Spring Cloud Stream.

(2) In the case of multiple consumers, Pubsub will try to distribute messages with the same Key to one messager to ensure order. This makes it sometimes difficult for other consumers to pick up the message and consume it in the case of Auto Scale, which can be controlled by configuring its size.

Five other

5.1 monitor

GCP has a mature Monitoring suite, Cloud Monitoring, which we can use directly. You can see how many messages are sent, how many messages are waiting to be consumed, and so on.

5.2 Consumers automatically expand capacity

If the consumer’s processing speed is too slow, it can be increased to solve the problem. The scheme is to automatically expand according to the number of messages retained in Pubsub. There are two solutions, one using Keda and the other using Cloud Monitor. Both are similar, taking the size of the queue and then scaling it flexibly through Kubernetes’ HPA.

Kubernetes uses Keda for flexible scaling and better Resource Utilization

6 summarizes

Pubsub is relatively easy to use, as you only write about producers and consumers. But there are a lot of details, so we’ll talk about them later.

Welcome to pay attention to the wechat public number “Pumpkin slow Talk”, will continue to update for you…

Read more and share more; Write more. Organize more.