Monday, 23 September 2013

Installing a Storm Cluster

From the previous post, you must have clear idea, what Storm is meant for. Now the next step is to setup the Storm cluster on the machine.

Following are the prerequisites for setting the  cluster :
  •  Linux Operating system
  • Java 6 installed
  • Python installed

Installation Steps :
Following steps are needed to get a Storm Cluster up and running.

  • Set up Zookeeper ClusterZookeeper is used as a coordinator  in Storm cluster.  You can refer here to see the installation steps for Zookeeper.
  • Install native dependencies on Nimbus & Worker machine  There are some dependencies which are required by Storm i.e. ZeroMQ, JZMQ.  These native dependencies are needed only on Storm cluster. While using Storm in local mode, Storm uses a pure Java Messaging system so you don’t need to install native dependencies there.  But in cluster mode, it’s needed.     
            ZeroMQ 2.1.7 Installation : Storm has been tested with ZeroMQ 2.1.7, and this is the recommended ZeroMQ release that    you install. You can download it from hereFollowing are the steps for installation.


tar –xzf zeromq-2.1.7.tar.gz

cd zeromq-2.1.7

./configure

make

sudo make install

JZMQ Installation: JZMQ is the java binding ZeroMQ.  Here are the steps:


cd jzmq

./autogen.sh

./configure

make

sudo make install

  • Download Storm release from here  and copy it to every machine in cluster (nimbus and worker machine.
  • Configure  storm.yaml Storm release contains a file at the conf/storm.yaml  with the default configuration to run the Storm Daemon.
storm.zookeeper.servers:
     - "IP_MACHINE_1"
     - "IP_MACHINE_2"
storm.local.dir: "/home/hadoop/STORM_DIR"
java.library.path: "/usr/local/lib"
nimbus.host: "IP_OF_NIMBUS_MACHINE"
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

               Copy this storm.yaml into “~/.storm/” location.
  • Now launch all the Storm Daemons.
hduser@ubuntu:/usr/local/storm-0.8.1$ bin/storm nimbus &
hduser@ubuntu:/usr/local/storm-0.8.1$ bin/storm supervisor &
hduser@ubuntu:/usr/local/storm-0.8.1$ bin/storm ui &

Here is the example to start with Storm Topology: 
                 https://github.com/nathanmarz/storm-starter

1 comment:

  1. Thanks for sharing this informative information..
    Installing a Storm Cluster Following are the prerequisites for setting the cluster http://www.s4techno.com/blog/2016/08/13/installing-a-storm-cluster/

    ReplyDelete