Skip to main content

How to create Dynamic Kafka Topic Through Java

This is post is about how to create a Kafka topic dynamically through Java.In one of my project, we(me and my friend Jaya Ananthram) were required to create dynamic Kafka topic through Java. Since there is no documentation on Kafka official documentation we are struggled to create dynamic Kafka topic through Java. Then After a long search ,we found this solution for creating the Kafka topic through Java. 

import java.util.Properties;
import kafka.admin.AdminUtils;
import kafka.utils.ZKStringSerializer$;
import kafka.utils.ZkUtils;
import org.I0Itec.zkclient.ZkClient;
import org.I0Itec.zkclient.ZkConnection;

public class KafkaTopicCreationInJava
{
    public static void main(String[] args) throws Exception {
        ZkClient zkClient = null;
        ZkUtils zkUtils = null;
        try {
            String zookeeperHosts = "localhost:2181";  
            // If multiple zookeeper then -> String zookeeperHosts = "192.168.1.1:2181,192.168.1.2:2181";
            int sessionTimeOutInMs = 15 * 1000; // 15 secs
            int connectionTimeOutInMs = 10 * 1000; // 10 secs

            zkClient = new ZkClient(zookeeperHosts, sessionTimeOutInMs, connectionTimeOutInMs, ZKStringSerializer$.MODULE$);
            zkUtils = new ZkUtils(zkClient, new ZkConnection(zookeeperHosts), false);

            String topicName = "testTopic";
            int noOfPartitions = 2;
            int noOfReplication = 3;
            Properties topicConfiguration = new Properties();

            AdminUtils.createTopic(zkUtils, topicName, noOfPartitions, noOfReplication, topicConfiguration);

        } catch (Exception ex) {
            ex.printStackTrace();
        } finally {
            if (zkClient != null) {
                zkClient.close();
            }
        }
    }
}
I am assuming both Zookeeper and Kafka are configured with default configuration at localhost and the service running. While creating ZkClient object, we must set ZkSerializer as ZKStringSerializer$.MODULE$, So that ZkClient will create a topic at Kafka. If we didn't mention ZkSerializer, the topic will not be properly created at Kafka. It will just create topic directories at Zookeepers and It will show that topic is created successfully. But by the time we try to push tuples, it will fail and throw Topic is not exists.

Note: The ZKStringSerializer$.MODULE$ will return the singleton ZKStringSerializer object.

Comments

  1. Hi Harish,
    I am working on similar project but I want to know what if security is enabled in kafka,where and how do you provide authentication detail in this code?

    zkUtils = new ZkUtils(zkClient, new ZkConnection(zookeeperHosts), false);

    here you have security false.

    thanks
    Vijay Shekhawat

    ReplyDelete

Post a Comment

Popular posts from this blog

Install Hadoop 2.5.1 on Windows 7 - 64Bit Operating System

This post is about installing Single Node Cluster Hadoop 2.5.1 (latest stable version) on Windows 7 Operating Systems.Hadoop was primarily designed for Linux platform. Hadoop supports for windows from its version 2.2, but we need prepare our platform binaries. Hadoop official website recommend Windows developers to use this build for development environment and not on production, since it is not completely tested success on Windows platform. This post describes the procedure for generating the Hadoop build for Windows platform. Generating Hadoop Build For Windows Platform Step 1:Install Microsoft Windows SDK 7.1 In my case, I have used Windows 7 64 bit Operating System. Download Microsoft Windows SDK 7.1 from Microsoft Official website and install it. While installing Windows SDK,I have faced problem like C++ 2010 Redistribution is already installed. This problem will happen only if we have installed C++ 2010 Redistribution of higher version compared to the Windows SDK. ...

Install Spring Tool Suite on Ubuntu

This post is about installing Spring Tool Suite (STS) on Ubuntu. The Spring Tool Suite is an Eclipse-based development environment that is customized for developing Spring applications. Step 1: Download the latest Spring Tool Suite for Linux from STS official website: http://spring.io/tools/sts/all  Step 2: Extract into any folder which you prefer. My extracted Spring Tool Suite locations is /home/harishshan/springsource Step 3: Create the Menu icon for quick access sudo nano /usr/share/applications/STS.desktop Step 4: Enter the following content [Desktop Entry] Name=SpringSource Tool Suite Comment=SpringSource Tool Suite Exec=/home/harishshan/springsource/sts-3.4.0-RELEASE/STS Icon=/home/harishshan/springsource/sts-3.4.0-RELEASE/icon.xpm StartupNotify=true Terminal=false Type=Application Categories=Development;IDE;Java; Step 5: Now you can check from Quick Menu by typing " Spring "

How to fix Kindle wrong time left in chapter book

The most of kindle reader use the time left in chapter or time left in book options provided at left bottom of the page to estimate completing the book or the current chapter. But sometimes it was not  accurate and displaying 5 min left in chapter instead of 30 min left in chapter. To fix this issue just type ;ReadingTimeReset at search field and enter. Kindly refer the photo for the same.