The authors are Yu Liu, Documentation engineer of StreamNative, and Yong Zhang, software engineer.

About the Apache Pulsar

Apache Pulsar is the top project of Apache Software Foundation. It is the next generation cloud native distributed message flow platform, integrating message, storage and lightweight functional computing. It adopts the architecture design of computing and storage separation, supports multi-tenant, persistent storage, and multi-room cross-region data replication. It has strong consistency, high throughput, low latency, and high scalability. GitHub address: github.com/apache/puls…

Two months after Apache Pulsar 2.6.3 was released, Apache Pulsar 2.7.1 was released on March 18, 2021.

Apache Pulsar 2.7.1 adds many optimizations and fixes a number of bugs, Covering Broker, Proxy, Pulsar Perf, Transaction, Pulsar Admin, Pulsar SQL, Client, Function, Pulsar IO, and Tiered Storage, Further enrich and improve Apache Pulsar’s capabilities as a cloud native streaming data platform.

Apache Pulsar 2.7.1 received about 717 commits from the community, and 210 PR from the consolidated community. More and more commits are becoming contributor members in Pulsar. Many of the code and documentation contributions come from Chinese developers, and China is becoming more and more powerful.

Here’s a detailed look at some of the major optimizations and bug fixes in Apache Pulsar 2.7.1.

Broker

Optimize schema comparison logic

Prior to 2.7.1, Pulsar Schema compared schema types by converting schemas to bytes. When a schema is uploaded, if it contains Spaces or newlines, the schema types are incompatible. In version 2.7.1, Pulsar Schema uses equals to compare schema types, so there is no schema type incompatibility.

For more information, see PR-9612.

Send an alarm before token expires

Before 2.7.1, when the token was about to expire, Pulsar had no relevant alarm, the client with token authentication enabled had no relevant indicator to query its connection status, and could not determine whether the token expired caused the problem. Pulsar in 2.7.1 provides expiredTokenMetrics and expiringTokenMinutesMetrics indicators, monitoring state of token will expire.

For more information, see PR-9321.

Fixed system Topics not being created automatically

In 2.7.0, if automatic creation of a topic is turned off, system topics under the corresponding namespace cannot be automatically created. Topic Policy functions depend on System Topic, so Topic Policy cannot be used. In 2.7.1, System topics are not controlled by the automatic creation of topics.

For more information, see PR-9272.

Fixed an error message when getting information about a nonexistent topic partition

When using pulsar-admin to retrieve partition information for a nonexistent topic, the broker returns 0 without any error message. In 2.7.1, when getting a topic that does not exist, Pulsar will report an error and tell you that the topic does not exist.

For more information, see PR-8818.

Support for removing late messages when purging the Topic backlog

Prior to 2.7.1, if the Topic backlog was cleared when sending delayed messages, delayed messages were not cleared. 2.7.1 Fixed the problem.

For more information, see PR-8691.

Proxy

Repair thebindAddressProblems that do not take effect

Prior to 2.7.1, bindAddress worked for webPort, but not servicePort. 2.7.1 Fixed this issue so that bindAddress works for both ports.

For more information, see PR-9068.

Fixed inaccurate authentication error information

When the Proxy sends an error to the client, it returns all error types to the client as ServiceNotReady. The client converts the error into a LookupError to handle the error. As a result, the client receives incorrect error information. And cause the client to fail (because LookupError is a retried error, the client is always reconnecting). 2.7.1 This problem is fixed. Accurate error information is returned to the client and normal error processing is ensured.

For more information, see PR-9055.

Supports the HAProxy proxy protocol

Typically, Pulsar uses a Pulsar Proxy as a proxy to connect to clients and process requests. 2.7.1 Supports HAProxy, which is more general.

For more information, see PR-8686.

Client

Support for Pulsar Perf to get JVM information

In 2.7.1, Pulsar Perf supports retrieving JVM information, including Netty Max Memory, JVM Command line arguments, and Actual Max heap size, This facilitates debugging using Pulsar Perf.

For more information, see PR-9769.

Fixed an issue where Pulsar Admin expire-message command failed without notification

The EXPIRE Message command fails when a request or subscription sent with an EXPIRE Message has caught up with a message that is about to expire and the user is not notified. 2.7.1 Fixed the issue of throwing an exception when the command failed.

For more information, see PR-9561.

Adds original information to a message in a dead letter queue

When a message is sent to a dead-letter queue, the “original information” (for example, message ID and topic) is lost. If there is a problem, this is not conducive to finding the source of the problem. 2.7.1 Added more “original information” for messages added to the dead-letter queue to facilitate troubleshooting in case of problems.

For more information, see PR-9655.

Supports end-to-end encryption on Python clients

2.7.1 Added the Python client end-to-end encryption function to improve message security.

For more information, see PR-9588.

Support for setting subscription names for C++ reader

When using C++ reader, the default subscription name is randomly generated and cannot be specified. 2.7.1 Supports subscription names for readers.

For more information, see PR-8823.

Function

Fixed an issue where the download directory could not be defined using K8S Runtime

Prior to 2.7.1, K8S runtime did not correctly use the downloadDirectory defined by the user in the configuration file, which resulted in an inability to define the downloadDirectory. 2.7.1 Fixed this problem.

For more information, see PR-9377.

Fixed an issue where log information was not correctly output to log Topic

Prior to 2.7.1, different Context objects in Java Function prevented log information from being correctly generated into the corresponding log Topic, making it impossible for users to obtain log information. 2.7.1 Fixed the problem.

For more information, see PR-9299.

Fixed compatibility of authentication configurations between Broker and function

In the broker and function configuration brokerClientAuthenticationEnabled at the same time, can lead to the function the Worker cannot be started. 2.7.1 Fixed the problem.

For more information, see PR-9190.

Support for exposing schema information in source Topic to downstream sinks

2.7.1 Support sink can obtain the schema of messages sent by source and continue processing in Sink.

For more information, see PR-8854.

The reference information

  • Download Apache Pulsar 2.7.1.

  • Check out the complete Apache Pulsar 2.7.1 release notes.

  • View the PR list for Apache Pulsar 2.7.1.

  • Join the Apache Pulsar Slack Channel.

  • Subscribe to the Apache Pulsar mailing list.

We look forward to your joining us and contributing to the thriving growth of Apache Pulsar!