Grading Project 2

You can grade project 2 interactively or by recording a video of your demo. Regardless of how you choose to demo the project, here are the steps:

  1. Start your cluster on orion (12 storage nodes, 12 computation nodes)
  2. Store your dataset (show that you can store files and have the system respect the line endings – highlight this in code if necessary).
    • The file should be big enough that it will span all 12 computation nodes so we can fully exercise the system.
      • Consequently, the jobs we run will use 12 mappers.
  3. Launch jobs on the dataset
    • First, run your word count job
    • Next, the log analyzer
  4. Show the load balancing decisions that were made when choosing where to run the job (showing the terminal output is fine). This should include both the mappers and reducers.
    • You should be able to configure a different number of reducers for the two jobs. Each reducer will write its output to your DFS.
      • Use a single reducer for one job, and, for example, 4 reducers for another.
  5. Show job progress output
  6. Show job output (retrieve from your DFS)
    • Show that outputs are sorted correctly
  7. Provide a walkthrough of the MapReduce job code

Finally, make sure I have access to your