Assignment 2
ITCS-4010/5010: Cloud Computing for Data Analysis, Spring 2009

Assignment 2:

• The due date for Assignment 2 is 11:59:59pm on Thursday, February 19 .

• Assignment 2 description: assign2.pdf .

• Sample data files: roadnot.txt by Robert Frost, daffodils.txt by William Wordsworth, cagedbird.txt by Maya Angelou.

• These two lines can help you get the filename.
  FileSplit fileSplit = (FileSplit) reporter.getInputSplit();
  String fileName = fileSplit.getPath().getName();

• Example output format for inverted index program: index_all.txt and full inverted index program: fullindex_all.txt when run on all input files. (Note that these output files have been truncated.)

• Please use the following naming convention for the source code you submit.
  Simple inverted index: Index.java
  Query on simple inverted index: QueryIndex.java
  Full inverted index: FullIndex.java
  Query on full inverted index: QueryFullIndex.java

• Please use the following naming convention for the output files you submit. If output is generated by the inverted index program on a single input file, name the file "index_<inputfilename>.txt". For example, inverted index output for input file "roadnot.txt" should be "index_roadnot.txt". If output is generated by the inverted index program on all input files, name the file "index_all.txt". Name the output generated by the full inverted index program on all input files "fullindex_all.txt".

• Assignment 2 submission: Please submit using Blackboard. You should zip your output files (following naming convention above), your source code for the inverted index program and full inverted index program, and any README files so we can run your code.

  Note: The assignment is due by Thursday, February 19, 11:59:59pm. You can submit your assignment multiple times. However only the most recent submission will be graded. The timestamp on your most recent submission will also be used to compute late days used.

• We recommend you use Hadoop 0.19.0, JDK 1.6.x, Eclipse 3.3 (or higher).
• Follow one of the following tutorials, based on your computing environment:
  1. Yahoo Hadoop tutorial: If you want to use virtual machine image running Ubuntu Linux
  2. Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)
  3. Hadoop Quick Start (for single node installation), and Hadoop on Windows XP (single-node cluster, using cygwin)
• Hadoop MapReduce tutorial: Has explanatory details of the Hadoop framework

• Hadoop
• Hadoop Wiki
• MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat, OSDI'04: Sixth Symposium on Operating System Design and Implementation, San Francisco, December 2004.
• MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat, Communications of the ACM, Vol. 51, No. 1, page 107-113, January 2008.
• The Google File System, Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung, Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP 2003), Lake George, NY, October, 2003.

1. Java SE Development Kit (JDK) 6 , (links for Windows are here)
2. Ubuntu Linux
3. Cygwin
4. Eclipse and Java tutorial
5. Eclipse MapReduce plugin

Back to Cloud Computing course page