6 responses to “Run The Latest Whirr And Deploy HBase In Minutes”

  1. Lars George

    Hi Phil,

    Great post! We are still working on a few kinks but once Whirr 0.3.0 is out and will be released with CDH it will be even easier to get going as no build is required (obviously).

    A few notes, you are missing the “tt” (i.e TaskTracker) in the explanation table for the cluster template. Also, just to reiterate, all of the above services share the same instance. So in your example you will start 1+5 EC2 server with the various services running on them.

    Finally, what is also cool is that Whirr creates a local “$HOME/.whirr//” directory, so in your example $HOME/.whirr/hbase/ which has various files to help you working with the cluster. One of those is the “hadoop-proxy.sh” (start it like “source $HOME/.whirr/hbase/hadoop-proxy.sh &” to launch it in the background) which sets up the SOCKS proxy for you so that you can talk to the servers and access the web UI for the master and region servers.

    There is also a “instances” which lists nicely the servers in the cluster, their local and remote IPs and roles. It could be used by a script or program to parse the details and be able to communicate with the servers.

    Thanks again for writing this post!


  2. Lynn Monson

    Very helpful. The only thing I tripped on was understanding how the local keypair file relates to Whirr. In my case, I followed your tutorial from a clean machine, itself running on EC2. Since that machine had no ~/.ssh/id_rsa file, Whirr failed.

    It’s still not completely clear to me how Whirr bootstraps the machines, but I did learn that eventually Whirr uploads your local public key to the bootstrapped instances. So a simple invocation of ssh-keygen -t rsa got everything running.

    Thanks again for the time and effort spent on this tutorial.

  3. Eugene Koontz

    Thanks a lot for this post; I found that your instructions work great, but if you prefer to use git, you can do :

    git clone git://git.apache.org/whirr.git

    instead of:

    svn co http://svn.apache.org/repos/asf/incubator/whirr/trunk whirr-trunk

  4. Tim

    Great instructions!
    I believe the /incubator is not needed in the svn URL

    When mine starts, it says:

    Completed configuration of hbase
    Web UI available at
    Wrote HBase site file /Users/tim/.whirr/hbase/hbase-site.xml
    Wrote HBase proxy script /Users/tim/.whirr/hbase/hbase-proxy.sh
    Completed configuration of hbase role hadoop-datanode
    Completed configuration of hbase role hadoop-tasktracker
    Starting to run scripts on cluster for phase start on instances: us-east-1/i-116dab74
    Running start phase script on: us-east-1/i-116dab74
    start phase script run completed on: us-east-1/i-116dab74
    Successfully executed start script: [output=, error=, exitCode=0]
    Finished running start phase scripts on all cluster instances
    Started cluster of 4 instances

    but it seems only the zookeeper actually started up and there is no JT or HBase master:

    tim@ip-10-79-37-92:~$ ps -ef | grep java
    root 3214 1 0 13:29 ? 00:00:00 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/local/zookeeper-3.3.3/bin/../build/classes:/usr/local/zookeeper-3.3.3/bin/../build/lib/*.jar:/usr/local/zookeeper-3.3.3/bin/../zookeeper-3.3.3.jar:/usr/local/zookeeper-3.3.3/bin/../lib/log4j-1.2.15.jar:/usr/local/zookeeper-3.3.3/bin/../lib/jline-0.9.94.jar:/usr/local/zookeeper-3.3.3/bin/../src/java/lib/*.jar:/etc/zookeeper/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg

    All the slave node services started it seems.

  5. Kiko Aumond

    This is an extremely useful post, thank you so much for writing it!