Release Notes -kafka -version 2.7.0

Below is a summary of the JIRA issues addressed in Kafka’s 2.7.0 release. See the Kafka project website for complete documentation, getting started, and project information about this release.

Note about upgrades: Before upgrading your cluster, check the upgrade documentation for that version. The upgrade notes discuss any key information about incompatibilities and breakthrough changes, performance changes, and any other changes that may affect your Kafka production deployment.

The latest version of the document can be in kafka.apache.org/documentati… To find it.

New features

  • [Kafka-9893] – Configurable TCP connection timeout and improved initial metadata acquisition
  • [kafka-9915] – Controls creating a theme, creating a partition, and deleting a theme
  • [Kafka-10054] – Adds TRACE level end-to-end latency metrics for streams
  • [kafka-10259] -KIP-554: Adds the BROker-side SCRAM configuration API
  • [kafka-10338] – Supports SSL certificates and private keys in PEM format

To improve the

  • [Kafka-5636] – Added sliding window support for aggregation

  • [Kafka-6585] – Integrate repetitive logic on reset tools

  • [Kafka-6733] – Supports printing the additional ConsumerRecord field in DefaultMessageFormatter

  • [kafka-7334] – It is recommended to change the configuration of state.dir when FileNotFoundException occurs.

  • [Kafka-8360] – RequestQueueSize JMX metric is not specified.

  • [Kafka-8836] – Add protocol between brokers to change ISR

  • [kafka-9018] – Kafka Connect – throws clearer exceptions in case of serialization errors

  • [Kafka-9076] – MirrorMaker 2.0 automates consumer offset synchronization

  • [Kafka-9194] – Missing document for replicaMaxWaitTimeMs configuration value

  • [kafka-9439] – Added more public API tests for KafkaProducer

  • [Kafka-9450] – Decouple internal state refresh from commit

  • [Kafka-9587] – The producer configuration is omitted from the documentation

  • [kafka-9924] – Adds the RocksDB memory consumption to the RocksDB indicator

  • [Kafka-9929] – Support for reverse iterators on WindowStore

  • [Kafka-10002] – Improves StopReplicaRequest performance when a large number of partitions need to be deleted

  • [kafka-10020] -kip-616: Renames the Serdes instance implied in Kafka-streams scala

  • [Kafka-10023] – Enforces connection creation rates for the entire broker and per listener (KIP-612, Part 1).

  • [kafka – 10033] – if you change the theme configuration does not exist, AdminClient should throw UnknownTopicOrPartitionException rather than UnknownServerException.

  • [kafka – 10035] – AbstractResetIntegrationTest improvement

  • [kafka – 10040] – let PreferredReplicaImbalanceCount index calculation more efficient

  • [kafka-10077] – Filter results for false tombstones downstream of the state-store

  • [kafka – 10098] – delete SaslAuthenticatorTest. Unnecessary escaping in Java regular expressions.

  • [kafka-10106] – Record the processing time of LeaderAndIsr requests

  • [Kafka-10122] – Consumers should also be allowed to beat during rebalancing.

  • [Kafka-10126] – Discard the unused option in ConsumerPerformance.

  • [Kafka-10131] – Minimize the use of — ZooKeeper markers in ducktape testing.

  • [kafka-10135] – Extract Task#executeAndMaybeSwallow as a generic tool.

  • [kafka-10135] – Extract Task#executeAndMaybeSwallow and make it a generic function of TaskManager.

  • [kafka-10141] – Add more details to delete information for log segments

  • [Kafka-10143] – Added integration tests to ensure that the redistribution tool can change replication quotas during rebalancing

  • [kafka-10168] – Rename the public StreamsConfig variable

  • [Kafka-10186] – Transactions that terminate pending data should throw a non-fatal exception

  • [Kafka-10205] – NullPointerException in StreamTask

  • [Kafka-10211] – Add DirectoryConfigProvider

  • [Kafka-10225] – Increases the default ZK session timeout for system tests

  • [Kafka-10232] – MirrorMaker2 Internal theme formatting

  • [Kafka-10277] – Relax the non-empty key requirement for kstream-GlobalkTable connections.

  • [Kafka-10279] – Allows dynamic updating of certificates with additional SubjectAltNames

  • [kafka-10305] – Print usage when parsing ConsumerPerformance fails

  • [Kafka-10316] – Consider renaming the getter method for interactive queries

  • [Kafka-10355] – Accidental removal of source topics is treated as a special fault

  • [Kafka-10367] – Allows you to run Streams demo applications using configuration files

  • [kafka-10403] – Use Java collections instead of Scala collections when generating MBeans properties.

  • [kafka-10407] – Add linger. Ms parameter support for KafkaLog4jAppender

  • [kafka-10418] – Incomplete error/documentation when changing topic configuration over Kafka-topics with –bootstrap-server

  • [Kafka-10458] – Requires a way to update the quota of tokenbuckets registered with the Sensor.

  • [Kafka-10459] – Document IQ API where the order between stores does not hold

  • [Kafka-10478] – Advertising. Listeners should allow duplicate ports

  • [Kafka-10494] – Stream: If the node itself is materialized, enableSendingOldValues should not call the parent node.

  • [Kafka-10503] – MockProducer does not throw a ClassCastException when the subject is not partitioned.

  • [Kafka-10509] – Add metrics to track throttling time due to the connection rate quota being touched

  • [kafka-10521] – Remove ZK watch after partition reallocation

  • [kafka-10584] – IndexSearchType should use a sealed trait instead of an Enumeration

  • [Kafka-10611] – Merge log errors to avoid double errors

  • [Kafka-10801] – Documents about configuration use the same HTML anchor tags in several places

  • [Kafka-10802] – False log information on consumer startup

Bug

  • [kafka – 6078] – survey ReassignPartitionsClusterTest. ShouldExpandCluster failure.

  • [Kafka-7970] – Lack of topics causes services to shut down without exception

  • [kafka-8033] – flip-flop test PlaintextConsumerTest#testFetchInvalidOffset

  • [kafka – 8098] – obvious test AdminClientIntegrationTest# testConsumerGroups

  • ] [kafka – 8264 – odd test PlaintextConsumerTest# testLowMaxFetchSizeForRequestAndPartition

  • [kafka-8334] – Occasionally an OffsetCommit timeout occurs

  • [Kafka-8362] – LogCleaner gets stuck when partitions between log directories are moved.

  • [Kafka-8630] – A ClassCastException occurred while unit testing a convection processor using WindowStore.

  • [kafka-9066] – Kafka Connect JMX: The source and sink indicators of a failed task are lost

  • [Kafka-9381] – Javadocs + Scaladocs not released in Maven center

  • [Kafka-9514] – Protocol generator generates useless conditions when a field is set to null and uses flexible versions

  • [kafka – 9516] – swing test PlaintextProducerSendTest# testNonBlockingProducer

  • [Kafka-9584] – Deleting header files causes concurrent modification exceptions.

  • [Kafka-9666] – Transactional producer’s Epoch cannot be reset

  • [Kafka-9769] -Replicamanager Partition. MakeFollower Adds LeaderEpoch when ZooKeeper is disconnected

  • [kafka-9845] -plugin. path property does not work with config Provider.

  • [kafka-10014] – always try to close all channels in Selector#close

  • [Kafka-10048] – With MM2, consumers may experience a data gap after failover

  • [Kafka-10079] – Improves thread level stickiness for stateful tasks

  • [kafka-10109] – kafka-acl. Sh /AclCommand Opens multiple AdminClients

  • [kafka-10120] -describelogdirsresult exposes inner classes

  • [Kafka-10133] – Messages in the destination cluster cannot be compressed with MM2

  • [Kafka-10134] – After upgrading to 2.5, Kafka consumers are experiencing high CPU issues when rebalancing

  • [Kafka-10140] – Incremental configuration API does not cover plug-in configuration changes

  • [kafka-10147] – MockAdminClient#describeConfigs(Collection

    Unable to process broker resources.

  • [kafka-10158] – Repair the scintillation kafka. Admin. TopicCommandWithAdminClientTest# testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress.

  • [kafka-10160] – Kafka MM2 Consumer configuration

  • [Kafka-10179] – State Store passes the wrong Changelog topic to the Serde to get the optimized source table

  • [Kafka-10180] – TLSv1.3 system tests should not be run under Java 8

  • [kafka-10188] – The sink task preCommit method is called after the task is stopped

  • [Kafka-10189] – Reset metric EventQueueTimeMs

  • [kafka-10192] – wavering test BlockingConnectorTest#testBlockInConnectorStop

  • [kafka-10193] – Add priority to controller events with callbacks

  • [Kafka-10200] – MockProcessorContext does not work with WindowStores

  • [Kafka-10215] – MockProcessorContext does not work with SessionStores

  • [kafka-10218] – DistributedHerder’s canReadConfigs field is never reset to true.

  • [kafka-10220] – The NPE used to describe a resource

  • [Kafka-10224] – Incorrect license terms for jerseys

  • [kafka-10235] – Fix flashing transactions_test.py

  • [Kafka-10240] – Sink tasks should not throw WakeupException on shutdown

  • [kafka – 10243] – ConcurrentModificationException when handling connection timeout

  • [kafka-10268] – Dynamic configurations like “–delete-config log.retention. Ms “do not work.

  • [Kafka-10271] – Performance degradation when obtaining a key from a partition

  • [kafka-10272] – kafka-server-stop.sh Failed on IBM I.

  • [kafka-10282] – If a log is quickly deleted and recreated, log metrics are deleted

  • [Kafka-10284] – Group member updates due to static member rejoining should be persisted

  • [Kafka-10300] – Fix erratic core/group_mode_transactions_test.py

  • [kafka – 10312] – MetadataCache. GetPartitionMetadata leadership of the partition is not MetadataCache may return not correct the error code.

  • [kafka-10314] – Reassign KafkaStorageException when the offline log directory exists

  • [kafka – 10321] – shouldDieOnInvalidOffsetExceptionDuringStartup on JDK11 blocked forever.

  • [Kafka-10326] – Both the serializer and the desserializer should be able to see the generated client ID.

  • [Kafka-10332] – MirrorMaker2 cannot detect a remote topic if it was created first.

  • [Kafka-10340] – The source connector should report an error when attempting to generate a record for a non-existent topic, instead of suspending forever.

  • [Kafka-10362] – When restoring Streams activity with EOS, checkpoint files should be deleted.

  • [kafka-10384] – Separate the converter from the generated message

  • [Kafka-10387] – The SMT configuration cannot be included with a source connector that uses the topic.creation.* property.

  • [Kafka-10391] – Streams should override checkpoints to exclude corrupted partitions

  • [Kafka-10395] – TopologyTestDriver does not work in dynamic topic routing

  • [kafka-10401] – GroupMetadataManager Ignores the current_state_TIMESTAMP field of GROUP_METADATA_VALUE_SCHEMA_V3.

  • [kafka-10426] – The KafkaConfigBackingStore has a deadlock problem

  • [Kafka-10439] -Connect Values class loses accuracy for integers larger than 64 bits

  • [kafka-10454] – A Kafka flow gets stuck in an infinite REBALANCING cycle when the flow <> table connection partitions do not match

  • [Kafka-10455] – Probe rebalancing is not guaranteed to be triggered by a non-lead member

  • [kafka-10463] – The necessary tools in Dockerfile should include Git

  • [kafka-10477] – The Sink Connector has a DataException when attempting to convert the empty key Kafka record to a connection record.

  • [Kafka-10479] – Throws an exception if the user tries to update the configuration of an existing listener.

  • [Kafka-10502] – Null cannot be set locally to the thread as it may cause a memory leak.

  • [kafka-10515] -NPE: If the application is distributed, the foreign key connection serde may not be initialized to the default serDE.

  • [kafka-10520] – InitProducerId may be blocked if the least loaded node is not ready to send.

  • [kafka-10531] – KafkaBasedLog can hibernate negative values

  • [Kafka-10557] – Documentation missing when describing topic configuration, including documentation

  • [Kafka-10559] – During internal topic validation, do not close the entire application when a TimeoutException occurs

  • [Kafka-10564] – Keep track of the situation regarding the removal of obsolete status directories

  • [Kafka-10574] – Infinite loop in SimpleHeaderConverter and Values classes

  • [kafka-10598] – KafkaStreams reports inappropriate error messages for IQ

  • [Kafka-10600] – If a connector does not define a property, Connect will add an error for that property to the validation result.

  • [Kafka-10602] – DLQ Reporter throws NPE when reporting from different threads.

  • [Kafka-10606] – Automatically creates non-existent topics when fetching metadata for all topics

  • [kafka-10613] – If the version of the list-offset request is <4, the Broker should not set the leader epoch

  • [Kafka-10616] – StreamThread is “IllegalStateException: Processor is closed”

  • [kafka-10631] -producerfencedexception not processed on Offest Commit

  • [kafka – 10638] – due to the more strict inspection of storage QueryableStateIntegrationTest failure.

  • [Kafka-10647] – Owned partitions are serialized only if the consumer protocol version >=0

  • [Kafka-10651] – The assignor reports offsets from uninitialized tasks

  • [kafka – 10664] – Streams cannot cover damaged offsets, cause an infinite OffsetOutOfRangeException cycle.

  • [Kafka-10687] – Production requests should be collided for new error code PRODUCE_FENCED.

  • [kafka-10755] – Commit delays should be taken into account when calculating the next commit timestamp.

  • [kafka-10758] – When a new subject is added, the Kafka stream consumed from the mode goes to PENDING_SHUTDOWN.

  • [kafka-10799] – AlterIsr path does not update ISR shrink/expand tables.

  • [kafka – 10811] – system exit MirrorConnectorsIntegrationTest# testReplication

  • [kafka-10813] -StreamsProducer should capture InvalidProducerEpoch and throw TaskMigrated in all cases.

# # task

  • [kafka – 10068] – validation HighAvailabilityTaskAssignor performance under large cluster and topology
  • [Kafka-10084] – System test failed. StreamsEosTest.test_failure_and_recovery_complex
  • [Kafka-10153] – Error report in connection document
  • [Kafka-10341] – Added version 2.6 for stream and system tests
  • [kafka-10456] – Kafka-console-producer.sh Error description in the help
  • [kafka-10556] – NPE occurs if sasL. Mechanism is not identified.
  • [kafka-10592] – System tests cannot run after python3 merge
  • [kafka-10669] – ListOffsetRequest: make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field to 1

# # test

  • [kafka – 9273] – refactoring AbstractJoinIntegrationTest and subclasses
  • [Kafka-9943] – Enable tlSV.1.3 “Run all” execution in system tests.
  • [Kafka-10191] – Fix flickering StreamsOptimizedTest
  • [kafka – 10255] – repair fragile testOneWayReplicationWithAutorOffsetSync1 test
  • [Kafka-10286] – Connection system testing should wait for workers to join the team
  • [kafka – 10295] – ConnectDistributedTest. Stop test_bounce should wait for grace.
  • [kafka-10351] – Adds missing IOExceptions test to GlobalStateManagerImpl
  • [kafka-10404] – Flaky Test kafka.api.SaslSslConsumerTest.testCoordinatorFailover

The subtasks

  • [Kafka-9432] – Replace DescribeConfigs request/response with automatic protocol
  • [Kafka-9434] – Replace AlterReplicaLogDirs request/response with automatic protocol
  • [kafka-9627] – Replace ListOffset request/response with automatic protocol
  • [kafka-9629] – Replace Fetch request/response with automatic protocol
  • [Kafka-9911] – Implementation of new producer fence error
  • [Kafka-10027] – The read path to implement the functional version plan
  • [kafka-10028] – Write path to implement the feature version plan
  • [kafka-10042] – Make INVALID_PRODUCER_EPOCH abortable from Produce response.
  • [Kafka-10163] – Implement broker changes
  • [Kafka-10164] – Implement administrator changes
  • [kafka-10174] – The –bootstrap-server ducktape test is preferred using kafka_configs.sh
  • [kafka-10213] – Preferred in the Ducktape test for the Kafka client — Bootstrap-server
  • [kafka-10258] – Get rid of the USe_zk_connection flag in the public methods in kafka-py.
  • [Kafka-10261] – Introduce kap-478 processor with spacer
  • [Kafka-10270] – Add a broker to the controller channel manager to redirect AlterConfig
  • [kafka-10308] – Fix fragile core/round_trip_fault_test.py
  • [Kafka-10319] – Fix unknown offset sum on Trunk
  • [kafka-10379] – StreamBuilder#addGlobalStore() for kip-478
  • [kafka-10435] – Changes the obtain protocol for KIP-595
  • [Kafka-10437] -KIP-478: Implement test – Utility change
  • [Kafka-10447] – Migrate tool modules to JUnit 5
  • [kafka-10516] – Implement Topic Command changes
  • [kafka-10535] -KIP-478: Implement StateStoreContext and Record
  • [kafka-10536] -kip-478: Implement the KStream change
  • [kafka-10562] -KIP-478: delegate the storage wrapper to the new init method
  • [kafka-10570] – Rename kiP-629’s JMXReporter configuration
  • [Kafka-10571] – Replace blackout in KIE-629 with backoff
  • [kafka-10572] – Rename the MirrorMaker 2 blacklist configuration for KIP-629.
  • [kafka-10573] – Rename kiP-629’s connection conversion configuration
  • [Kafka-10599] – Basic CLI tool for implementing functional version systems
  • [Kafka-10605] -KIP-478: Repel the replaced processor API member