Project 1 Test Files

The test files for Project 1 are stored in orion01:/bigdata/mmalensek/project1. Here’s a list of their names and MD5 checksums:

• test_1.txt (95 bytes) - 8563385c0f993accd25f40a278a1f7b4
• test_2.bin (1 MB) - f3c9a2169881d9e49d4951cd76db35f9
• test_3.bin (10 MB) - 6ddb07a82b92bdf80c7523a66e58fedc
• test_4.bin (150 MB) - 850d1ea7e205d7a519b05b89693d859e
• test_5.jpg (1.2 GB) - b1ab0804049d0dd659ca8c471587145c
• test_6 (6.7 GB) - db86f795e1d70cf8c71814205352566a
• test_7 (18 GB) - 956d159b9139598efcd34ed4bbb3b8f7

We should be able to store the file, retrieve it, and verify that the checksum matches those listed above. You can use the md5sum utility on Linux machines to test this.

Occasionally the grader may decide to generate new files for you to use, with similar properties to those listed above.

Setup

To get ready for grading, you will need to launch (BEFORE grading begins):

• The Controller
• 12 storage nodes
• A terminal window to run your client from

Set your storage directory to somewhere under /bigdata/students/$(whoami)/.

You should be able to show your heartbeat messages are working, then we’ll proceed with the test cases.

The grader will log into one of the orion machines, compile your project, and run your client. PLEASE make sure you have instructions so the grader can easily figure out how to use your client application.

Test Cases

1. Storing a few files that map to different amounts of chunks. We should be able to specify the chunk size.
2. Retrieving the files and verifying md5sum.
3. Attempt to store a file that already exists. (Should be rejected).
4. Delete a file. Try to retrieve it after deletion.
5. List files in the system
6. Kill a random storage node, make sure the failure is detected and replication level maintained. Please make sure you have a clear way of demonstrating this.
7. Retrieve files that were impacted by the failure.
8. Corrupt chunk data on disk
9. Retrieve file related to that chunk, verify file is restored. Please make sure you have a clear way of demonstrating this.

Code Walkthrough Examples

• Client -> Controller communication
• Client -> Storage Node communication
• Controller replication logic and fault detection
• Storage node corrupt file detection
• Show .proto files