Apache Kafka InvalidReplicationFactorException

When creating a topic this Exception may be thrown by Kafka.

Scenario

If we try to create a topic with replication factor larger than the number of cluster, we will see an error.

Exception

This is the exception thrown by Apache Kafka.

	Error while executing topic command : Replication factor: 5 larger than available brokers: 2.
	[2018-01-23 18:02:37,534] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException:
	Replication factor: 5 larger than ava
	ilable brokers: 2.
	(kafka.admin.TopicCommand$)

view raw gistfile1.txt hosted with ❤ by GitHub

Resolution

Ensure that the replication factor is less than or equal to the number of brokers in the cluster.

Kafka Exception - kafka.common.InconsistentBrokerIdException

I came across this exception when I was working with Apache Kafka.

Scenario

Set up multiple Kafka brokers in a cluster. Copy the server.properties to create a new propeties file for the new broker. I was following the instructions here https://kafka.apache.org/documentation/#quickstart_multibroker

Result

The first broker started without error. When I started the second broker, I saw this exception - kafka.common.InconsistentBrokerIdException.

	[2018-01-23 17:30:19,004] FATAL Fatal error during KafkaServer startup. Prepare
	to shutdown (kafka.server.KafkaServer)
	kafka.common.InconsistentBrokerIdException: Configured broker.id 2 doesn't match
	stored broker.id 1 in meta.properties. If you moved your data, make sure your c
	onfigured broker.id matches. If you intend to create a new broker, you should re
	move all data in your data directories (log.dirs).
	at kafka.server.KafkaServer.getBrokerIdAndOfflineDirs(KafkaServer.scala:
	615)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:
	38)
	at kafka.Kafka$.main(Kafka.scala:92)
	at kafka.Kafka.main(Kafka.scala)
	[2018-01-23 17:30:19,007] INFO shutting down (kafka.server.KafkaServer)
	[2018-01-23 17:30:19,010] INFO Terminate ZkClient event thread. (org.I0Itec.zkcl
	ient.ZkEventThread)
	[2018-01-23 17:30:19,017] INFO Session: 0x16125c42cd00004 closed (org.apache.zoo
	keeper.ZooKeeper)
	[2018-01-23 17:30:19,018] INFO EventThread shut down for session: 0x16125c42cd00
	004 (org.apache.zookeeper.ClientCnxn)
	[2018-01-23 17:30:19,020] INFO shut down completed (kafka.server.KafkaServer)
	[2018-01-23 17:30:19,022] FATAL Exiting Kafka. (kafka.server.KafkaServerStartabl
	e)
	[2018-01-23 17:30:19,024] INFO shutting down (kafka.server.KafkaServer)

view raw .java hosted with ❤ by GitHub

Resolution

I found that the cause was that when I copied the proerties file I had missed changing the "log.dirs" property. If the two brokers share the same "log.dirs" property then we get this error. I restarted the second instance with a unique value for "log.dirs" property. The issue was resolved.