Consuming Kafka's internal consumer offsets topic

Since version 0.8.2, kafka has had the ability to store consumer offsets in an internal compacted topic called __consumer_offsets

If you are interested in viewing the consumer offsets stored on the __consumer_offsets, you should do the following.

  • Create a small config with the exclude.internal.topics property set to false
  • Use the OffsetsMessageFormatter class as the formatter
    • For Kafka 0.8.2.x, it is kafka.server.OffsetManager\$OffsetsMessageFormatter
    • For Kafka 0.9.x.x and 0.10.0.0, it is kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter
  • If you want to consume all consumer offsets from the beginning as opposed to just the latest, you’ll need to add --from-beginning flag to the kafka-console-consumer.sh script

For Kafka 0.8.2.x

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Only consume the latest consumer offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" \
--zookeeper localhost:2181 --topic __consumer_offsets

For Kafka 0.9.x.x and 0.10.0.0

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Only consume the latest consumer offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" \
--zookeeper localhost:2181 --topic __consumer_offsets

Notes

  • From version 0.8.2, the old consumers will by default commit their offsets to zookeeper. This can be changed to storing on the __consumer_offsets topic by setting the consumer config property offsets.storage=kafka. The new consumers commit their offsets to the __consumer_offsets
  • For Kafka 0.9.x.x and above, you might notice that you’ve consumed more messages, from __consumer_offsets topic, than were displayed. This is because the formatter has been changed to only work on Offset messages

Dayo Oliyide

Read more posts by this author.