Grading Project 2
You can grade project 2 interactively or by recording a video of your demo. Regardless of how you choose to demo the project, here are the steps:
- Start your cluster on orion (12 storage nodes, 12 computation nodes)
- Store your dataset (show that you can store files and have the system respect the line endings – highlight this in code if necessary).
- The file should be big enough that it will span all 12 computation nodes so we can fully exercise the system.
- Consequently, the jobs we run will use 12 mappers.
- The file should be big enough that it will span all 12 computation nodes so we can fully exercise the system.
- Launch jobs on the dataset
- First, run your word count job
- Next, the log analyzer
- Show the load balancing decisions that were made when choosing where to run the job (showing the terminal output is fine). This should include both the mappers and reducers.
- You should be able to configure a different number of reducers for the two jobs. Each reducer will write its output to your DFS.
- Use a single reducer for one job, and, for example, 4 reducers for another.
- You should be able to configure a different number of reducers for the two jobs. Each reducer will write its output to your DFS.
- Show job progress output
- Show job output (retrieve from your DFS)
- Show that outputs are sorted correctly
- Provide a walkthrough of the MapReduce job code
Finally, make sure I have access to your
- Design document
- Retrospective
- Video link (if applicable)