Migrate brokers by modifying broker IDs in meta.properties

  • The environment

  • Broker migration Steps

    • Step-1: Start a new Broker as the old one.
    • Step-2: Stops the new Broker and the old Broker it is replacing.
    • Step 3: in the Cloudera Manager and data directory/meta properties will be a new change for old Broker’s Brokerbroker.id.
    • Step-4: Start a new Broker

The purpose of this paper:

Our goal is to replace Kafka Broker nodes. We have 2 new nodes added to the CDP PvC Base cluster. We migrated the two Kafka brokers we had previously used to these two new nodes. Refer to the ClouderaCDP official documentation for guidance.

The environment

The following Kafka Brokers are currently in use:

Broker Host IP
1546345005 c3669-node3.coelab.cloudera.com 172.25.36.133
1546345009 c3669-node4.coelab.cloudera.com 172.25.36.207
1546345001 c3669-node2.coelab.cloudera.com 172.25.36.206

Broker migration Steps

Step-1: Start a new Broker as the old one.

CM UI -> Kafka -> Instances -> Add Role Instances

We are adding brokers one at a time, and we need to replace brokers one by one.

After host-10-17-102-138.coe.cloudera.com is added as a Kafka Broker role, it should not start immediately. We should now see in CM that the Kafka cluster is in a stale information state, so we deploy the client configuration file using instructions from the CM UI.

Now start the newly added Broker at host-10-17-102-138.coe.cloudera.com.

Step-2: Stops the new Broker and the old Broker it is replacing.

In this case, I use the new Broker Broker – 1546345005-1546351866 to replace the old, the old at c3669-node3.coelab.cloudera.com, New at host-10-17-102-138.coe.cloudera.com.

Step 3: in the Cloudera Manager and data directory/meta properties will be a new change for old Broker’s Brokerbroker.id.

We need to change broker.id to the ID of the old Broker on the new Broker host.

We also need to change the broker.id on the CM UI, which also causes stale information calculations, so deploy the client configuration file as CM prompts.

Step-4: Start a new Broker.

We can now launch a new Broker at host-10-17-102-138.coe.cloudera.com on CM UI. Before startup, we should see an alarm about the “temper Replicas” health check.

After starting a new Broker, after some time (depending on the amount of data on the cluster), the Kafka Broker cluster will be healthy again.

This is what Broker replacement/migration is all about. Thanks for reading.