Just a small question , what if I wanna connect to the instance , where I can find the key in order to connect to it.

Cheers,

Joao

We have following properties set :

whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 jt+nn,1 dn+tt
whirr.provider=ec2
whirr.credential=mYar/KSbx+UL+nqGr9hSgGHIOqXC9tjNcuO9UwF/
whirr.identity=AKIAJCDTYGREJYIECQZA
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop-install-runurl=cloudera/cdh/install
whirr.hadoop-configure-runurl=cloudera/cdh/post-configure
whirr.image-id=ami-3bc9997e
whirr.hardware-id=i-bffa23f8
whirr.location-id=us.west-1c

But we are getting following error:

[ec2-user@ip-10-170-103-243 ~]$ ./whirr-0.3.0-cdh3u1/bin/whirr launch-cluster –config hadoop.properties –run-url-base http://whirr.s3.amazonaws.com/0.3.0-cdh3u0/util
Bootstrapping cluster
Configuring template
Exception in thread “main” java.util.NoSuchElementException
at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:147)
at com.google.common.collect.Iterators.find(Iterators.java:679)
at com.google.common.collect.Iterables.find(Iterables.java:555)
at org.jclouds.compute.domain.internal.TemplateBuilderImpl.locationId(TemplateBuilderImpl.java:492)
at org.apache.whirr.service.jclouds.TemplateBuilderStrategy.configureTemplateBuilder(TemplateBuilderStrategy.java:41)
at org.apache.whirr.service.hadoop.HadoopTemplateBuilderStrategy.configureTemplateBuilder(HadoopTemplateBuilderStrategy.java:31)
at org.apache.whirr.cluster.actions.BootstrapClusterAction.buildTemplate(BootstrapClusterAction.java:144)
at org.apache.whirr.cluster.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:94)
at org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:74)
at org.apache.whirr.service.Service.launchCluster(Service.java:71)
at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:65)
at org.apache.whirr.cli.Main.main(Main.java:91)

Can we get any help ? What shold we do ?

Thanks,
S.Sarkar

I am using Hadoop Mapreduce to predict secondary structure of a given long sequence. The idea is, I have a chunks of segments of a sequence and they are written into a single file input where each line is one segment. I have used one of the programs for secondary structure predictions as my mapper code (Hadoop Streaming).
The out put of the mapper was successful that it produces the predicted structures in terms of dot-bracket notation. I want to use a simple reducer that glue all the outputs from the mapper in an orderly manner.
For Example, If my input was like

….
And my mapper output is a predicted structure but not in order

What I am looking is a reducer code that sorts and Glue and outputs in a form similar to the following:
……

Any help…Thanks

Do you mind re-posting this to whirr’s mailing list?

http://incubator.apache.org/whirr/mail-lists.html

There’s a release almost finished which should support cdh much better (v0.6)

puts prev_key + separator + key_total.to_s

and it worked.

2. I followed the instruction to set up Hadoop client in the the host initiating whirr and got the following error when it tried to connect to name node.
11/08/07 01:03:48 INFO ipc.Client: Retrying connect to server: ec2-107-20-60-75.
compute-1.amazonaws.com/10.116.78.198:8020. Already tried 9 time(s).
Bad connection to FS. command aborted. exception: Call to ec2-107-20-60-75.compu
te-1.amazonaws.com/10.116.78.198:8020 failed on local exception: java.net.Socket
Exception: Connection refused

It has to do with security. I checked the security group jclouds#myhadoopcluster3#us-east-1. It allows inbound on 80, 50070, 50030 only from the host initiating whirr launch-cluster and allow inbound on 8020, 8021 only from the name node host. I added rules to allow inbound on 8020, 8021 from the host initiating whirr and apply the rule change. That doesn’t help. In my case, the host initiating whirr launch-cluster is a EC2 instance too.

3. I can ssh to cluster hosts from the host initiating whirr without any key. iptable is empty and selinux is disabled. Network rules seems set up outside the linux box. No luck.

4. I ends up transferring files to name nodes and run map reduce job there. Whirr script create /user/hive/warehouse but no /user/ec2-user. Need to create that directory and input sub-directory. You might also add -jobconf mapred.reduce.tasks=1 since the default is 10 in this case.

Thanks.

It worked well if my hadoop.properties as below:

whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 jt+nn,1 dn+tt
whirr.provider=ec2
whirr.identity=
whirr.credential=
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hardware-id= m1.large
whirr.location-id=us-west-1
whirr.hadoop-install-runurl=cloudera/cdh/install
whirr.hadoop-configure-runurl=cloudera/cdh/post-configure

But once i added two more lines into hadoop.properties file, it went wrong:
whirr.image-id= us-west-1/ami-***** (my ami)
jclouds.ec2.ami-owners=(my owner id>

I have posted a question on whirr forum and see if i could get any solution. Will update here if i get anything. thx

There are recipes in the latest version of whirr here, as well:
http://svn.apache.org/repos/asf/incubator/whirr/trunk/recipes/hadoop-ec2.properties

Unrelated, but if you don’t mind trying 0.5.0, voting it up can get the new rev, including recipes released!
http://people.apache.org/~tomwhite/whirr-0.5.0-incubating-candidate-1/

Cheers,
A

whirr.image-id= us-west-1/ami-**********
jclouds.ec2.ami-owners=******************
whirr.hardware-id= m1.large
whirr.location-id=us-west-1

But it doesn’t work. Any thought? thanks

Next question though… how do you use Cassandra for input/output (while still using Ruby)?

I know you can run Hadoop jobs against Cassandra in Java with the InputFormat they provide, but how to do so using streaming?

http://www.quora.com/What-are-some-good-toy-problems-in-data-science

and play some more.

Not all is well though – I received a bill from AWS for $100 this morning! While following your tutorial I fumbled my way through signing up for AWS and assumed that I would only ever be operating within a free tier – boy was I wrong! Some three weeks later and I realise that I’ve had “small” EC2 servers going all this time …

So, may I suggest that you please augment your post to direct the reader to shut down (and terminate) their EC2 instances after they are done with the tute, and, perhaps, include directions on how to ensure that they use Amazon’s free http://aws.amazon.com/free/ tier?

It is evident that I’ve made some silly mistakes while following this but I hope I am not atypical of your garden variety newbie that is curious about big data. I hope that I find a way to get Amazon to reverse charges (don’t like my chances though). In any case, my enthusiasm is not dampened – thanks again for getting me started

Navin

Apologies – and thanks for taking the time to reply …

Navin

There’s probably an issue with the property syntax inside your hadoop.properties. Have a look at http://incubator.apache.org/projects/whirr.html and https://cwiki.apache.org/WHIRR/configuration-guide.html

If you still have issues, send your query to the user list, you’ll get on track quickly!

whirr-user@incubator.apache.org

Cheers,
-Adrian

[~/src/cloudera/whirr-0.1.0+23]$ bin/whirr launch-cluster --config hadoop.properties                                                                                                     rvm:ruby-1.8.7-p299
Launching myhadoopcluster cluster
Exception in thread "main" com.google.inject.CreationException: Guice creation errors:

1) No implementation for java.lang.String annotated with @com.google.inject.name.Named(value=jclouds.credential) was bound.
  while locating java.lang.String annotated with @com.google.inject.name.Named(value=jclouds.credential)
    for parameter 2 at org.jclouds.aws.filters.FormSigner.(FormSigner.java:91)
  at org.jclouds.aws.config.AWSFormSigningRestClientModule.provideRequestSigner(AWSFormSigningRestClientModule.java:66)

1 error
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:410)
	at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:166)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:118)
	at com.google.inject.InjectorBuilder.build(InjectorBuilder.java:100)
	at com.google.inject.Guice.createInjector(Guice.java:95)
	at com.google.inject.Guice.createInjector(Guice.java:72)
	at org.jclouds.rest.RestContextBuilder.buildInjector(RestContextBuilder.java:141)
	at org.jclouds.compute.ComputeServiceContextBuilder.buildInjector(ComputeServiceContextBuilder.java:53)
	at org.jclouds.aws.ec2.EC2ContextBuilder.buildInjector(EC2ContextBuilder.java:101)
	at org.jclouds.compute.ComputeServiceContextBuilder.buildComputeServiceContext(ComputeServiceContextBuilder.java:66)
	at org.jclouds.compute.ComputeServiceContextFactory.buildContextUnwrappingExceptions(ComputeServiceContextFactory.java:72)
	at org.jclouds.compute.ComputeServiceContextFactory.createContext(ComputeServiceContextFactory.java:114)
	at org.apache.whirr.service.ComputeServiceContextBuilder.build(ComputeServiceContextBuilder.java:41)
	at org.apache.whirr.service.hadoop.HadoopService.launchCluster(HadoopService.java:84)
	at org.apache.whirr.service.hadoop.HadoopService.launchCluster(HadoopService.java:61)
	at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
	at org.apache.whirr.cli.Main.run(Main.java:65)
	at org.apache.whirr.cli.Main.main(Main.java:91)

PS: The rvm:ruby-1.8.7-p299 is part of my prompt and not in the output.

Thanks in advance for any pointers on getting this resolved.

Regards,
Navin

Check out this wiki page…
http://wiki.apache.org/hadoop/AmazonS3

This would make a good follow post, so keep watching this space!

one more question,

is there a way to use s3 buckets for the input and output?

In this example the Ruby scripts for Map and Reduce are in the same directory as the bash script that I created for running the job. They do not need to be. You just need to tell the job starter where to find them. For instance, for the map.rb script I have…

-file map.rb

This is a relative path, meaning it’s in the same directory. It could also be relative in another directory..

-file ../someotherdirectory/map.rb

or we could give an absolute path…

-file /Users/phil/someotherdirectory/map.rb

The bash script can also be written is any directory. The path to the JAR file is absolute, so the only other dependencies are the Map and Reduce scripts as I mentioned above.

I hope that answers your question.

I was able to follow along and everything worked (after correcting for my typos).

One thing that wasn’t clear, one can be in any dir for the ruby & bash file work. I had done this originally in the hadoop folder.

thanks