Apache Pulsar 2.6.1 was released on August 21, 2020, 2 months after the release of Apache Pulsar 2.6.0.
In Pulsar 2.6.1 version, there are 102 committers from the community, and more and more partners start to participate in Pulsar community construction and become contributors. Let’s take a look at what’s new in version 2.6.1.
Broker related improvements
📣 limits the batch size to the minimum of “maxNumberOfMessages” and “maxSizeOfMessages”
Prior to version 2.6.0, the minimum values of maxNumberOfMessages and maxSizeOfMessages in the BatchReceive policy did not affect the batch size. If the batch size is larger than the receiveQueue size set in consumer (suppose the batch size is 3000 and receiveQ is 500), the following problems may occur:
When using multi-topic mode in consumer, the client blocks and does not receive any messages. Even if a user sets a timeout policy in a batch, the client does not recover.
In version 2.6.1, we fixed this problem by setting the batch size to the minimum values in “maxNumberOfMessages” and “maxSizeOfMessages”.
For more details see PR-6865: github.com/apache/puls…
📣 Resolve hash range conflicts caused by using sticky hash range in Key_Shared
In previous versions, when users used “stickyHashRange” in the Key_Shared subscription model, the hash ranges specified by consumers were not allowed to overlap. For example, the hash range of consumer-1 is [[0,99], [400,65535]], and the hash range of consumer-2 is [[100,399]].
This is because the start and end positions in the Stick Hash range are not checked on the broker side. In normal cases, start cannot be greater than end. In version 2.6.1, we added a check mechanism to avoid hash range conflicts.
For more details see PR-7231: github.com/apache/puls…
📣 Rectify the error of obtaining the lookup permission
Currently, when Pulsar AuthorizationService checks lookup permission, having canProducer or canConsumer should have the ability to canLookup, but it does not. The code is as follows:
java
try {
return canLookupAsync(topicName, role, authenticationData)
.get(conf.getZooKeeperOperationTimeoutSeconds(), SECONDS);
}
Copy the codeIf the canProduce or canConsume methods throw an exception, canLookup only throws that exception and does not check for other permissions.
In version 2.6.1, canLookupAsync was used instead of the original behavior. See PR-7234: github.com/apache/puls for more details… .
📣 Repair topic that cannot be deleted when creating non-durable Cursor
NPE is returned when a nonpersistent cursor creation fails. Because the application continues to create subscription instances after an NPE occurs:
java
try {
cursor = ledger.newNonDurableCursor(startPosition, subscriptionName);
} catch (ManagedLedgerException e) {
subscriptionFuture.completeExceptionally(e);
}
return new PersistentSubscription(this, subscriptionName, cursor, false);
Copy the codeCauses the reference count for that topic to increase by one. When a user wants to delete this topic, the reference count is not cleared, so the topic cannot be deleted even with –force. In version 2.6.1, we solved the problem of not being able to delete a topic.
For more details see PR-7355: github.com/apache/puls…
📣 avoidManagedLedgerImpl.isOffloadedNeedsDeleteNPE occurs in the method
Before 2.6.1, the default value of offload-deletion-lag was null, which caused NPE problems. In version 2.6.1, we in ManagedLedgerImpl. IsOffloadedNeedsDelete method adding checks for a null value, avoid the problem.
For more details, see PR-7389: github.com/apache/puls… .
📣 Fixed an NPE issue when creating a new ledger that caused the producer to freeze
NPE is raised when ledger is created because network addresses cannot be resolved. If the NPE is raised before the timeout task is added, the timeout mechanism does not work. Unresolvable network addresses are common in Kubernetes environments. This can happen when the Bookie Pod or worker node is restarted.
In version 2.6.1, you can fix this problem by:
- Capture this NPE when creating a new ledger;
- When a timeout task is triggered, the callback is always executed. Because the callback can only fire once;
- Add a mechanism to detect whether the CreatingLedger status has changed.
For more details see PR-7401: github.com/apache/puls…
📣 Fixes NPE problems caused by using advertisedListeners
AdvertisedListeners = internal: pulsar: // node1:6650, external: pulsar: //node1.external: At 6650, the broker was unable to obtain ownership of the namespace package. If TLS is not enabled, we need to change BrokerServiceUrlTls.
For more details, see PR-7620: github.com/apache/puls… .
📣 The client incorrectly read the last entry. Procedure- 1This entry
Before version 2.6.1 getLargestBatchIndexWhenPossible () function has no return statement, when the entry is 1, the client will set the corresponding MessageData to the value of the current location, The value is sent to the client. When the client attempts to read the entry, the following problems occur:
779 [16:34:25. The pulsar - IO - 54-7: org. Apache. The bookkeeper. Client. LedgerHandle @ 748] the ERROR org.apache.bookkeeper.client.LedgerHandle - IncorrectParameterException on ledgerId:0 firstEntry:-1 lastEntry:-1 Pulsar 779 [16:34:25. - the client - IO - 82-1: org. Apache. The pulsar. Client. The impl. ConsumerImpl @ 1986] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://external-repl-prop/pulsar-function-admin/assignment][c-use-fw-localhost-0-function-assignment-initialize-r Eader-b21f7607c9] Successfully getLastMessageId 0:-1 16:34:25.779 [pulsar-client-io-82-1:org.apache.pulsar.client.impl.ClientCnx@602] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0 xc78f4a0e, L: / 127.0.0.1:55657 R: localhost / 127.0.0.1:55615] Received the error from the server: Failed to get batch size for entry org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input 16:34:25. 779 [pulsar - the client - IO - 82-1: org. Apache. The pulsar. The client. The impl. ClientCnx @ 612] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0 xc78f4a0e, L: / 127.0.0.1:55657 R: localhost / 127.0.0.1:55615] Received unknown request id from the server: 10Copy the codePr-7495 added a return statement to the code, GetLastEntry() reads the last entry instead of -1.
For more information see PR-7495: github.com/apache/puls…
ZooKeeper improvements
📣 Bookie rack-aware mapping using host names
UseHostName () and return false were added to PR-5607. This means that rack-style policies attempt to resolve the Bookie hostname to an IP address, and then use that IP address to determine which rack the Bookie belongs to.
This leads to two problems:
- The IP address does not match the host name recorded in the/bookies Z-node;
- An NPE exception is raised if an error occurs while resolving the bookie hostname (for example, transient DNS error); For the BookKeeper client, the Bookie has been unavailable in the cluster.
For example, line 77 in the following code throws an NPE because getAddress() gives a null and the address is not resolved:
java 74 if (dnsResolver.useHostName()) { 75 names.add(addr.getHostName()); 76 } else { 77 names.add(addr.getAddress().getHostAddress()); 78}Copy the codeBy default, dnsresolver.usehostname () returns true.
For more details see PR-7361: github.com/apache/puls…
Java Client related improvements
📣 fixed an issue where HTTP headers used in Athenz authentication could not be renamed
Athenz’s authentication plug-in allows the user to change the HTTP header name and send the authentication token to the proxy server with the roleHeader parameter. Changing the HTTP header name preserves the value of the “roleHeader” parameter on the “AuthenticationAthenz” side and uses it directly as the header name.
For more details, see PR-7311: github.com/apache/puls…
📣 Fixes a collection of batch Acks that are recycled multiple times
The root cause of multiple batch ACKS is that there are race conditions in batch ACK refresh and cumulative acknowledgement. Therefore, add a reclaim status check to the ackset to avoid recycling batch Acks multiple times.
For more details see PR-7409: github.com/apache/puls…
📣 Add a client that supports OAuth2 authentication
Pulsar supports client authentication using OAuth 2.0 access tokens. You can use tokens to identify Pulsar clients and associate them with certain principals (or roles) that allow certain actions (such as publishing to or consuming from a topic) to be performed.
This module directly supports OAuth 2.0’s Pulsar client authentication plug-in. After the client communicates with the OAuth 2.0 server, it obtains an access token from the OAuth 2.0 server and passes this access token to the Pulsar Broker for authentication.
As a result, the agent can still use the “pulsar org. Apache.. Broker. Authentication. AuthenticationProviderToken”, Users can also add their own AuthenticationProvider to use this module.
For more details, see PR-7420: github.com/apache/puls…
📣 no longer subscribes to this topic after consumer is closed
When the consumer reconnects to the broker, the competition condition is fixed in the consumer.
A race condition occurs when the consumer reconnects to the broker, and the connection is set to NULL when the consumer reconnects to the broker. If cosnumer is turned off at this point, the client no longer sends the shut down consumer command to the agent. Therefore, if the consumer reconnects to the broker, Consuemr will send the subscribe command again.
In version 2.6.1, when consumer’s connection is opened, consumer adds a status check. There is no need to send the subscribe command if the consumer is in the closed or closing state.
For more details, see PR-7589: github.com/apache/puls…
📣 OAuth2 authentication plug-in uses AsyncHttpClient
In previous versions, the OAuth2 client Auth plug-in made requests using the Apache HTTP client library, which was only used for hostname authentication. As described in PR-7612, AsyncHttpClient is used in version 2.6.1 to get rid of the Apache HTTP client library dependency. AsyncHttpClient is used elsewhere in the client and broker.
For more details, see PR-7615: github.com/apache/puls…
CPP Client related improvements
📣 Supports OAuth2 authentication on the CPP client
Pulsar supports client authentication using OAuth 2.0 access tokens. You can use tokens to identify Pulsar clients and associate them with certain principals (or roles) that allow them to perform certain actions, such as publishing to a topic or consuming from a topic.
In version 2.6.1, users are allowed to use OAuth2 authentication in CPP clients.
For more details, see PR-7467: github.com/apache/puls…
###📣 Fix partition index in closing callback
When callback is turned off in the partitioned producer/consumer, the partitioned index is always 0. We need to pass the ProducerImpl/ConsumerImpl internal Partition index field to the PartitionedProducerImpl/PartitionedConsumerImpl close callback.
For more details, see PR-7282: github.com/apache/puls…
📣 fixed segment crash caused by timer race status in C++ client
Before version 2.6.1, segment crashes occurred under race conditions:
- The close operation is called “keepaliveTimer_.reset ()”;
- At the same time,
startConsumerStatsTimerå’ŒhandleKeepAliveTimeoutMethod to access the timer.
In version 2.6.1, we fixed this issue and segment crashes no longer occur under race conditions.
For more details see PR-7572: github.com/apache/puls…
📣 supports reading credentials from files
Support for reading credentials from files, keeping them consistent with Java clients.
For more details, see PR-7606: github.com/apache/puls…
📣 fixed a segment error in many topic consumers when a connection error occurred
When there is an error in creating a consumer, the multi-themed consumer triggers a segmental error. This is due to the use of null callback to close some consumer calls.
In version 2.6.1, we fixed this issue.
For more details, see PR-7588: github.com/apache/puls…
Functions related improvement
📣 uses the standard hostname as the default for worker
Java 8 and Java 11 get host names differently. In Java 8, getHostName() returns the fully qualified hostname using the inetaddress.getLocalHost () argument. In Java 11, a simple host name is returned. With the getCanonicalHostName() ‘parameter, fully qualified hostname is returned in both Java 8 and Java 11.
For more details, see pr-7360 github.com/apache/puls…
📣 Fix backward compatibility introduced in 2.6.0
Pr-5985 breaks backward compatibility. If you run Function Worker and Broker separately, the following error occurs when the Function Worker and Broker are updated separately from 2.5 to 2.6:
text java.lang.NullPointerException: null\n\tat java.net.URI$Parser.parse(URI.java:3104) ~[?:?] java.net.URI.<init>(URI.java:600) ~[?:?]\n\tat java.net.URI.create(URI.java:881) ~[?:?] org.apache.pulsar.functions.worker.WorkerUtils.initializeDlogNamespace(WorkerUtils.java:160) Pulsar ~ [org. Apache. - the pulsar - functions provides - worker - 2.7.0 - the SNAPSHOT. Jar: 2.7.0 - the SNAPSHOT] org.apache.pulsar.functions.worker.Worker.initialize(Worker.java:155) Pulsar ~ [org. Apache. - the pulsar - functions provides - worker - 2.7.0 - the SNAPSHOT. Jar: 2.7.0 - the SNAPSHOT] org.apache.pulsar.functions.worker.Worker.start(Worker.java:69) Pulsar ~ [org. Apache. - the pulsar - functions provides - worker - 2.7.0 - the SNAPSHOT. Jar: 2.7.0 - the SNAPSHOT] org.apache.pulsar.functions.worker.FunctionWorkerStarter.main(FunctionWorkerStarter.java:67) [org. Apache. Pulsar - pulsar - functions provides - worker - 2.7.0 - the SNAPSHOT. Jar: 2.7.0 - the SNAPSHOT]Copy the codeError reason: version 2.5 broker will respond to a request for containing bookkeeperMetadataServiceUri fields, manage the client will return to the field to null, resulting in NPE.
In version 2.6.1, when initialization function when the worker, the value of BookkeeperMetadataServiceUri check to determine if it is null.
For more details, see PR-7528: github.com/apache/puls…
Pulsar Perf related improvements
📣 is supported in producer/ Consumer /reader of PulsAR-perftlsAllowInsecureConnection
Pulsar at the command line tools – perf support tlsAllowInsecureConnection configuration, to support the cluster of unsafe TLS connection producer/consumer/reader performance test.
For more details see PR-7300: github.com/apache/puls…
The reference information
- Apache Pulsar 2.6.1 website to download address: pulsar.apache.org/en/download…
- More about the Apache Pulsar 2.6.1 information, you can refer to below: pulsar.apache.org/release-not…
- Pulsar 2.6.1 PR list: github.com/apache/puls…